Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Gary Wilson <garypwilson@gmail.com>
Newsgroups: comp.lang.python.announce
Subject: [ANN] - Skyline v1.2.2-stable-luminosity
Date: Tue, 17 Jul 2018 21:37:28 +0100
Lines: 133
Approved: python-announce-list@python.org
Message-ID: <mailman.389.1531899369.2882.python-announce-list@python.org>
References: <CAPmkZmQwGzgLeegNcr2oq3cD-nfyZdAykZj0hTpj759=tKGh6w@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: list
Xref: csiph.com comp.lang.python.announce:2927

For those interested in anomaly detection and deflection in streamed
time series data.

I would like to announce a new release of Skyline v1.2.2 -
https://github.com/earthgecko/skyline/releases/tag/v1.2.2-stable-luminosity

What is Skyline?
----------------

Skyline v1.2.2 - For those interested in anomaly detection and deflection i=
n
streamed time series data.

Anomaly deflection.  The obvious next evolution in the use of all the anoma=
ly
detection data?

Skyline is a Python based anomaly detection/deflection stack that analyses,
anomaly detects, deflects, fingerprints and learns vast amounts of streamed
time series data.

- Skyline ingests streamed metric time series data - skyline/horizon
- Skyline uses a ```CONSENSUS``` of 3-sigma algorithms to detect anomalies =
on
  batch processed, streamed metric time series data - skyline/analyzer - an=
omaly
  detector
- It handles large and small seasonality in the data - skyline/mirage -
  anomaly deflector and detector
- You can train it on what is NOT anomalous and it learns - skyline/ionosph=
ere -
  anomaly deflector
- It records all your anomalies - skyline/panorama - anomaly memory
- It shows you all your data - skyline/webapp - anomaly view

Seeing as we desire our metrics to be not anomalous most of the time and we=
 want
to know when they ARE anomalous and given the fact that we try and build sy=
stems
that try to behave within not anomalous bounds so they perform well, due to
this we have:

- A lot of metric time series data that are not anomalous most of the time.
- A lot of data to train a system on what is NOT anomalous given a time ser=
ies
  data set, rather than simply focusing on what is anomalous, also focusing=
 on
  what is not anomalous.

To achieve this Skyline implements a novel time series similarities compari=
son
algorithm and a boundary layers methodology that generates fingerprints of =
time
series data using the sum of the values of features of the time series whic=
h
have been extracted using the tsfresh features extraction package -
https://github.com/blue-yonder/tsfresh and evaluation against boundary laye=
r
algorithms to determine whether a 3-sigma triggered anomaly is
actually a normal,
known pattern in the data.

The Skyline-Ionosphere-Tsfresh Time Series Similairities Comparison Algorit=
hm -
SITTSSCA first coined here :) compares the generated fingerprints of the tw=
o
time series and can determine if they closely resemble each other in terms =
of:

- of the amount of "power/energy", range and "movement" there is within the=
 time
  series data set somewhat like RMS - Erol Kalkan from United States
Geological Survey,
  =E2=80=9CAnother approach to compute the differences between two time ser=
ies is moving
  window root-mean-square. RMS can be run for both series separately. This =
way,
  you can compare the similarities in energy (gain) level of time series. Y=
ou
  may vary the window length for best resolution.=E2=80=9D
  (https://www.researchgate.net/post/How_can_I_perform_time_series_data_sim=
ilarity_measures_and_get_a_significance_level_p-value)
  http://stackoverflow.com/questions/5613244/root-mean-square-in-numpy-and-=
complications-of-matrix-and-arrays-of-numpy

The Skyline-Ionosphere-Tsfresh Time Series Similairities Comparison Algorit=
hm
compares how close the fingerprint values are as a percentage and
varying this percentage variable will either focusing the algorithm with gr=
eater
precision the closer to 0% the parameter gets, the perfect match (or possib=
ly
a mirror match too - unkonwn/untested) or it will incrementally increase th=
e
tolerance as the percentage variable increases and the matching will become
less and less reliable.

However there is a sweet spot and here SITTSSCA works extremely well :)

Added to SITTSSCA is an optional layer of simple boundary algorithms that a=
re
user defined during the operator training interaction with Skyline, where t=
he
operator augments the SITTSSCA results with boundaries that describe the
expected norm within the time series.  Very similar to being able to descri=
be
the Active Brownian Motion of a time series -
https://github.com/blue-yonder/tsfresh/pull/143#issuecomment-272314801

This results in an anomaly detection/deflection system which enables the us=
er to
very simply label time series and train Skyline on the peaks and troughs an=
d the
expected Active Brownian Motion or best effort thereof.

However it takes a little effort on your part to train Skyline, however wit=
h the
effort, Skyline is very good at doing anomaly detection and deflection.

With your help.  There is no easy anomaly detection or deflection, but ther=
e is
some reward with a bit of effort.

To learn more...
----------------
Project page -> https://github.com/earthgecko/skyline
Documentation -> https://earthgecko-skyline.readthedocs.io/en/latest/index.=
html

With the hope Skyline can make the universe a bit less anomalous.

Regards
Gary