Release 3.0.0

[nirgilboa]

Hi !

First off thank you for your work on project, it has been very useful :)
Have there been any issues reported on version 3.0.0-beta0 ?

Thanks,
Nir

[nsano-rururu]

With the docker image published on dockerhub, there seems to be no problem.
https://hub.docker.com/r/bitsensor/elastalert/

When I try to build a new Docker, I cannot create the Docker image properly with an error.
The content of Update to python3 and elastalert v0.2.4 # 174 in the pull request must be reflected in the Dockerfile.

Part of the message

Sending build context to Docker daemon  150.5kB
Step 1/29 : FROM alpine:latest as py-ea
 ---> a24bb4013296
Step 2/29 : ARG ELASTALERT_VERSION=v0.2.0b2
 ---> Using cache
 ---> 80c4512cc9b8
Step 3/29 : ENV ELASTALERT_VERSION=${ELASTALERT_VERSION}
 ---> Using cache
 ---> 635593853902
Step 4/29 : ARG ELASTALERT_URL=https://github.com/Yelp/elastalert/archive/$ELASTALERT_VERSION.zip
 ---> Using cache
 ---> 96eafc0ff511
Step 5/29 : ENV ELASTALERT_URL=${ELASTALERT_URL}
 ---> Using cache
 ---> 714703369180
Step 6/29 : ENV ELASTALERT_HOME /opt/elastalert
 ---> Using cache
 ---> a44f4cbd57a9
Step 7/29 : WORKDIR /opt
 ---> Using cache
 ---> 9b6ad7e8eb08
Step 8/29 : RUN apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget &&     wget -O elastalert.zip "${ELASTALERT_URL}" &&     unzip elastalert.zip &&     rm elastalert.zip &&     mv e* "${ELASTALERT_HOME}"
 ---> Running in 650fa41bdab3
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  py2-pip (missing):
    required by: world[py2-pip]
  py2-yaml (missing):
    required by: world[py2-yaml]
The command '/bin/sh -c apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget &&     wget -O elastalert.zip "${ELASTALERT_URL}" &&     unzip elastalert.zip &&     rm elastalert.zip &&     mv e* "${ELASTALERT_HOME}"' returned a non-zero code: 2

When I modify the Dockerfile with the pull requested content and build, I get some warnings.

Part of the message

npm WARN deprecated babel-preset-es2015@6.24.1: 🙌  Thanks for using Babel: we recommend using babel-preset-env now: please read https://babeljs.io/env to update!
npm WARN deprecated istanbul@0.4.5: This module is no longer maintained, try this instead:
npm WARN deprecated   npm i nyc
npm WARN deprecated Visit https://istanbul.js.org/integrations for other alternatives.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated joi@13.7.0: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated json3@3.3.2: Please use the native JSON object instead of JSON 3
npm WARN deprecated mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated chokidar@1.7.0: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated core-js@2.6.11: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN deprecated topo@3.0.3: This module has moved and is now available at @hapi/topo. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated hoek@5.0.4: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN deprecated hoek@6.1.3: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated circular-json@0.3.3: CircularJSON is in maintenance only, flatted is its successor.
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated

> dtrace-provider@0.8.8 install /opt/elastalert-server/node_modules/dtrace-provider
> node-gyp rebuild || node suppress-error.js

make: Entering directory '/opt/elastalert-server/node_modules/dtrace-provider/build'
  TOUCH Release/obj.target/DTraceProviderStub.stamp
make: Leaving directory '/opt/elastalert-server/node_modules/dtrace-provider/build'

> core-js@2.6.11 postinstall /opt/elastalert-server/node_modules/core-js
> node -e "try{require('./postinstall')}catch(e){}"

npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN ajv-keywords@2.1.1 requires a peer of ajv@^5.0.0 but none is installed. You must install peer dependencies yourself.

In order to reduce warnings, it has been confirmed that Elasticsearch 7.7.0 / Kibana 7.7.0 will alert slack by modifying the following files with library update,
node version change, Babel 6 to 7 upgrade ..

.nvmrc

「9.9.0」→「12.17.0」

12.17.0

index.js

Babel7 compatible
「require('babel-register');」→「require('@babel/register');」

require('@babel/register');
require('src');

.babelrc

Babel7 compatible

{
  "presets": [
    "@babel/preset-env"
  ]
}

package.json

Babel7 compatible
npm JavaScript library update

{
  "name": "@bitsensor/elastalert",
  "version": "3.0.0-beta.0",
  "description": "A server that runs ElastAlert and exposes REST API's for manipulating rules and alerts.",
  "license": "MIT",
  "main": "index.js",
  "author": {
    "name": "BitSensor",
    "url": "https://bitsensor.io",
    "email": "dev@bitsensor.io"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/bitsensor/elastalert.git"
  },
  "directories": {
    "lib": "./lib",
    "test": "./test"
  },
  "dependencies": {
    "@babel/cli": "^7.10.1",
    "@babel/core": "^7.10.1",
    "@babel/preset-env": "^7.10.1",
    "@babel/register": "^7.10.1",
    "body-parser": "^1.19.0",
    "bunyan": "^1.8.12",
    "cors": "^2.8.5",
    "elasticsearch": "^16.7.1",
    "express": "^4.17.1",
    "fs-extra": "^9.0.0",
    "joi": "^14.3.1",
    "lodash": "^4.17.15",
    "mkdirp": "^1.0.4",
    "object-resolve-path": "^1.1.1",
    "randomstring": "^1.1.5",
    "raven": "^2.6.4",
    "request": "^2.88.2",
    "request-promise-native": "^1.0.8",
    "tar": "^6.0.2",
    "ws": "^7.3.0"
  },
  "devDependencies": {
    "eslint": "^7.1.0",
    "husky": "^4.2.5",
    "istanbul": "^0.4.5",
    "mocha": "~7.2.0"
  },
  "scripts": {
    "build": "babel src -d lib",
    "start": "sh ./scripts/start.sh",
    "test": "./scripts/test.sh",
    "update-authors": "./scripts/update-authors.sh",
    "precommit": "./node_modules/eslint/bin/eslint.js ."
  }
}

Makefile

「v ?= v0.2.0b2」→「v ?= v0.2.4」
「docker pull alpine:latest && docker pull node:alpine」→「docker pull alpine:3.11 && docker pull node:14-alpine」
「docker run -it --rm -p 3030:3030 \」→「docker run -it --rm -p 3030:3030 -p 3333:3333 \」

v ?= v0.2.4

all: build

build:
	docker pull alpine:3.11 && docker pull node:14-alpine
	docker build --build-arg ELASTALERT_VERSION=$(v) -t elastalert .

server: build
	docker run -it --rm -p 3030:3030 -p 3333:3333 \
	--net="host" \
	elastalert:latest

.PHONY: build

docker image create

nvm install v12.17.0
nvm use
docker build -t elastalert:0.2.4 .

[nsano-rururu]

The following information is likely to require further action.

istanbul

stanbul is Deprecated.
https://www.npmjs.com/package/istanbul

This package has been deprecated
Author message:

This module is no longer maintained, try this instead: npm i >nyc Visit https://istanbul.js.org/integrations for other >alternatives.

＜Files that may be relevant＞
・.gitignore
・.npmignore
・package.json
・scripts/test.sh

joi

joi is deprecated, so it needs to be @hapi/joi.

https://www.npmjs.com/package/joi

This package has been deprecated
Author message:

This module has moved and is now available at @hapi/joi. >Please update your dependencies as this version is no longer >maintained an may contain bugs and security issues.

@hapi/joi
https://www.npmjs.com/package/@hapi/joi

＜Files that may be relevant＞
・package.json
・src/common/config/schema.js
・src/common/config/server_config.js
・src/handlers/test/post.js

request and request-promise-native

It became Deprecated on February 11, 2020.

Alternative libraries to request #3143

It seems that I'm just using request-promise-native in src/controllers/rules/index.js.

[nsano-rururu]

changed to @hapi/joi as joi is deprecated

package.json

delete
"joi": "^14.3.1",

add
"@hapi/joi": "^17.1.1",

{
  "name": "@bitsensor/elastalert",
  "version": "3.0.0-beta.0",
  "description": "A server that runs ElastAlert and exposes REST API's for manipulating rules and alerts.",
  "license": "MIT",
  "main": "index.js",
  "author": {
    "name": "BitSensor",
    "url": "https://bitsensor.io",
    "email": "dev@bitsensor.io"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/bitsensor/elastalert.git"
  },
  "directories": {
    "lib": "./lib",
    "test": "./test"
  },
  "dependencies": {
    "@babel/cli": "^7.10.1",
    "@babel/core": "^7.10.1",
    "@babel/preset-env": "^7.10.1",
    "@babel/register": "^7.10.1",
    "body-parser": "^1.19.0",
    "bunyan": "^1.8.12",
    "cors": "^2.8.5",
    "elasticsearch": "^16.7.1",
    "express": "^4.17.1",
    "fs-extra": "^9.0.0",
    "@hapi/joi": "^17.1.1",
    "lodash": "^4.17.15",
    "mkdirp": "^0.5.5",
    "object-resolve-path": "^1.1.1",
    "randomstring": "^1.1.5",
    "raven": "^2.6.4",
    "request": "^2.88.2",
    "request-promise-native": "^1.0.8",
    "tar": "^6.0.2",
    "ws": "^7.3.0"
  },
  "devDependencies": {
    "eslint": "^7.1.0",
    "husky": "^4.2.5",
    "istanbul": "^0.4.5",
    "mocha": "~7.2.0"
  },
  "scripts": {
    "build": "babel src -d lib",
    "start": "sh ./scripts/start.sh",
    "test": "./scripts/test.sh",
    "update-authors": "./scripts/update-authors.sh",
    "precommit": "./node_modules/eslint/bin/eslint.js ."
  }
}

src/common/config/schema.js

import Joi from 'joi'; → import Joi from '@hapi/joi';

// Defines the config's Joi schema
import Joi from '@hapi/joi';

const schema = Joi.object().keys({
  'appName': Joi.string().default('elastalert-server'),
  'es_host': Joi.string().default('elastalert'),
  'es_port': Joi.number().default(9200),
  'writeback_index': Joi.string().default('elastalert_status'),
  'port': Joi.number().default(3030),
  'wsport': Joi.number().default(3333),
  'elastalertPath': Joi.string().default('/opt/elastalert'),
  'rulesPath': Joi.object().keys({
    'relative': Joi.boolean().default(true),
    'path': Joi.string().default('/rules')
  }).default(),
  'templatesPath': Joi.object().keys({
    'relative': Joi.boolean().default(true),
    'path': Joi.string().default('/rule_templates')
  }).default(),
  'dataPath': Joi.object().keys({
    'relative': Joi.boolean().default(true),
    'path': Joi.string().default('/server_data')
  }).default()
}).default();

export default schema;

src/common/config/server_config.js

delete
import Joi from 'joi';
change
this._jsonConfig = Joi.validate(jsonConfig, schema).value; → this._jsonConfig = schema.validate(JSON.parse(jsonConfig)).value;

import fs from 'fs';
import path from 'path';
import schema from './schema';
import resolvePath from 'object-resolve-path';
import Logger from '../logger';

// Config file relative from project root
const configFile = 'config/config.json';
const devConfigFile = 'config/config.dev.json';

const configPath = path.join(process.cwd(), configFile);
const devConfigPath = path.join(process.cwd(), devConfigFile);
const logger = new Logger('Config');

export default class ServerConfig {
  constructor() {
    // ready() callbacks
    this._waitList = [];

    // Actual config
    this._jsonConfig = null;
  }

  /**
   * Get a value from the config.
   *
   * @param {String} key The key to load the value from.
   * @returns {*}
   */
  get(key) {
    return resolvePath(this._jsonConfig, key);
  }

  /**
   * Register a callback to call when the config is ready loading.
   *
   * @param {Function} callback The callback to register.
   */
  ready(callback) {
    this._waitList.push(callback);
  }

  /**
   * Loads the config by reading the config file or falling back to defaults.
   *
   * @returns {Promise} Returns a promise which resolves when everything is done (as a promise would).
   */
  load() {

    //TODO: Watch config file for changes and reload
    const self = this;
    return new Promise(function (resolve) {
      self._getConfig().then(function (config) {
        self._validate(config);
        resolve();
      });
    }).then(function () {
      self._waitList.forEach(function (callback) {
        callback();
      });
    });
  }

  _getConfig() {
    const self = this;

    return new Promise(function (resolve) {
      self._fileExists(devConfigPath).then(function (devConfigFound) {

        // If a dev config was found read it, otherwise check for normal config
        if (devConfigFound) {
          self._readFile(devConfigPath)
            .then(function (config) {
              resolve(config);
            })
            .catch(function () {
              resolve({});
            });
        } else {
          logger.info('Proceeding to look for normal config file.');
          self._fileExists(configPath).then(function (configFound) {
            if (configFound) {
              self._readFile(configPath)
                .then(function (config) {
                  resolve(config);
                })
                .catch(function () {
                  resolve({});
                });
            } else {
              logger.info('Using default config.');
              // If no config was found, return empty object to load defaults
              resolve({});
            }
          });
        }
      });
    });
  }

  /**
   * Checks if the config file exists and we have reading permissions
   *
   * @returns {Promise} Promise returning true if the file was found and false otherwise.
   * @private
   */
  _fileExists(filePath) {
    return new Promise(function (resolve) {
      // Check if the config file exists and has reading permissions
      try {
        fs.access(filePath, fs.F_OK | fs.R_OK, function (error) {
          if (error) {
            if (error.errno === -2) {
              logger.info(`No ${path.basename(filePath)} file was found in ${filePath}.`);
            } else {
              logger.warn(`${filePath} can't be read because of reading permission problems. Falling back to default configuration.`);
            }
            resolve(false);
          } else {
            logger.info(`A config file was found in ${filePath}. Using that config.`);
            resolve(true);
          }
        });
      } catch (error) {
        logger.error('Error getting access information with fs using `fs.access`. Error:', error);
      }
    });
  }

  /**
   * Reads the config file.
   *
   * @returns {Promise} Promise returning the config if successfully read. Rejects if reading the config failed.
   * @private
   */
  _readFile(file) {
    return new Promise(function (resolve, reject) {
      fs.readFile(file, 'utf8', function (error, config) {
        if (error) {
          logger.warn(`Unable to read config file in (${file}). Using default configuration. Error: `, error);
          reject();
        } else {
          resolve(config);
        }
      });
    });
  }

  /**
   * Validate the config using the Joi schema.
   *
   * @param {Object} jsonConfig The config to validate.
   * @private
   */
  _validate(jsonConfig) {
    // Validate the JSON config
    try {
      this._jsonConfig = schema.validate(JSON.parse(jsonConfig)).value;
    } catch (error) {
      logger.error('The config in \'config/config.json\' is not a valid config configuration. Error: ', error);
    }
  }
}

src/handlers/test/post.js

import Joi from 'joi'; → import Joi from '@hapi/joi';
const validationResult = Joi.validate(request.body.options, optionsSchema); → const validationResult = optionsSchema.validate(request.body.options);

import RouteLogger from '../../routes/route_logger';
import {sendRequestError} from '../../common/errors/utils';
import {BodyNotSendError, RuleNotSendError, OptionsInvalidError} from '../../common/errors/test_request_errors';
import Joi from '@hapi/joi';

let logger = new RouteLogger('/test', 'POST');

const optionsSchema = Joi.object().keys({
  testType: Joi.string().valid('all', 'schemaOnly', 'countOnly').default('all'),
  days: Joi.number().min(1).default(1),
  alert: Joi.boolean().default(false),
  format: Joi.string().default(''),
  maxResults: Joi.number().default(0)
}).default();

function analyzeRequest(request) {
  if (!request.body) {
    return new BodyNotSendError();
  }

  if (!request.body.rule) {
    return new RuleNotSendError();
  }

  const validationResult = optionsSchema.validate(request.body.options);

  if (validationResult.error) {
    return new OptionsInvalidError(validationResult.error);
  }

  let body = request.body;
  body.options = validationResult.value;

  return body;
}

export default function testPostHandler(request, response) {
  /**
   * @type {ElastalertServer}
   */
  let server = request.app.get('server');
  let body = analyzeRequest(request);

  if (body.error) {
    logger.sendFailed(body.error);
    sendRequestError(response, body.error);
  }

  server.testController.testRule(body.rule, body.options)
    .then(function (consoleOutput) {
      response.send(consoleOutput);
    })
    .catch(function (consoleOutput) {
      response.status(500).send(consoleOutput);
    });
}

[nsano-rururu]

istanbul

stanbulis Deprecated

Since there is no test code, coverage tool is not required at this time.
It would have been introduced and left in place to add.

Delete File

scripts/test.sh

package.json

delete：「"test": "./scripts/test.sh",」
delete：「"istanbul": "^0.4.5",」

{
  "name": "@bitsensor/elastalert",
  "version": "3.0.0-beta.0",
  "description": "A server that runs ElastAlert and exposes REST API's for manipulating rules and alerts.",
  "license": "MIT",
  "main": "index.js",
  "author": {
    "name": "BitSensor",
    "url": "https://bitsensor.io",
    "email": "dev@bitsensor.io"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/bitsensor/elastalert.git"
  },
  "directories": {
    "lib": "./lib",
    "test": "./test"
  },
  "dependencies": {
    "@babel/cli": "^7.10.1",
    "@babel/core": "^7.10.1",
    "@babel/preset-env": "^7.10.1",
    "@babel/register": "^7.10.1",
    "body-parser": "^1.19.0",
    "bunyan": "^1.8.12",
    "cors": "^2.8.5",
    "elasticsearch": "^16.7.1",
    "express": "^4.17.1",
    "fs-extra": "^9.0.0",
    "@hapi/joi": "^17.1.1",
    "lodash": "^4.17.15",
    "mkdirp": "^1.0.4",
    "object-resolve-path": "^1.1.1",
    "randomstring": "^1.1.5",
    "raven": "^2.6.4",
    "request": "^2.88.2",
    "request-promise-native": "^1.0.8",
    "tar": "^6.0.2",
    "ws": "^7.3.0"
  },
  "devDependencies": {
    "eslint": "^7.1.0",
    "husky": "^4.2.5",
    "mocha": "~7.2.0"
  },
  "scripts": {
    "build": "babel src -d lib",
    "start": "sh ./scripts/start.sh",
    "update-authors": "./scripts/update-authors.sh",
    "precommit": "./node_modules/eslint/bin/eslint.js ."
  }
}

[nirgilboa]

@nsano-rururu I'm thinking about forking and starting to maintain this on my own - would you like to make a joint effort?

[nirgilboa]

@nsano-rururu I'm thinking about forking and starting to maintain this on my own - would you like to make a joint effort?

Yes.

Cool - I also dropped an email to the maintainer, let's give him a day or two to respond

[nirgilboa]

Ok bitsensor apparently went under and revoked rights for this repo, so we have to continue on a fork.
Looks like two other contributors already forked and made some changes - let's pick one of the forks and pool our resources on it ?

[nirgilboa]

The repo which you mentioned diverges quite a bit from the master here.. not sure what you meant - do you think that fork is what we should be maintaining ?