Using PHP with Apache Kafka

pache Kafka is a fast, real-time, distributed, fault-tolerant message broker.

Using Kafka, you can transfer streaming data to the cluster, which is generated continuously, for example, history of website visits, financial transactions, online shopping orders, application logs, etc. This information can help to understand what is happening with the data right now, create recommendations, use machine learning or aggregate data for further analysis. All this takes seconds or minutes, instead of hours and days.

You can read more in the official documentation.

1. Download the code

First, download and untar the code

wget http://mirror.linux-ia64.org/apache/kafka/2.1.0/kafka_2.11-2.1.0.tgz
tar -xzf kafka_2.11-2.1.0.tgz
cd kafka_2.11-2.1.0

2. Start the server

Kafka uses ZooKeeper, we will use the script packaged with Kafka to get a single-node ZooKeeper instance.

bin/zookeeper-server-start.sh config/zookeeper.properties

Start the kafka server.

bin/kafka-server-start.sh config/server.properties

Now we can create a topic named “test” with a single partition and one replica.

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

3. Install PHP client

There are several php clients for Kafka. I recommend using this one https://github.com/weiboad/kafka-php, because I already used it in production and it showed better performance than some other php clients.

composer require nmred/kafka-php

4. Create consumer

Create file Consumer.php and set broker list to 127.0.0.1:9092 as we have Kafka cluster locally with one broker (node).

<?php
require '../vendor/autoload.php';

$config = \Kafka\ConsumerConfig::getInstance();
$config->setMetadataRefreshIntervalMs(10000);
$config->setMetadataBrokerList('127.0.0.1:9092');
$config->setGroupId('test');
$config->setBrokerVersion('1.0.0');
$config->setTopics(['test']);
$consumer = new \Kafka\Consumer();

$consumer->start(function($topic, $part, $message) {
    var_dump($message);
});

5. Create producer

A producer can work in two modes – asynchronous and synchronous. Let’s try both.

First, create file Producer.php with following content for async mode.

<?php
require '../vendor/autoload.php';

$config = \Kafka\ProducerConfig::getInstance();
$config->setMetadataRefreshIntervalMs(10000);
$config->setMetadataBrokerList('127.0.0.1:9092');
$config->setBrokerVersion('1.0.0');
$config->setRequiredAck(1);
$config->setIsAsyn(false);
$config->setProduceInterval(500);

$producer = new \Kafka\Producer(
    function() {
        return [
            [
                'topic' => 'test',
                'value' => 'test....message.',
                'key' => 'testkey',
            ],
        ];
    }
);

$producer->success(function($result) {
    var_dump($result);
});
$producer->error(function($errorCode) {
    var_dump($errorCode);
});
$producer->send(true);

And create file ProducerSync.php for sync mode where we send 100 messages to Kafka in a cycle.

<?php
require '../vendor/autoload.php';

$config = \Kafka\ProducerConfig::getInstance();
$config->setMetadataRefreshIntervalMs(10000);
$config->setMetadataBrokerList('127.0.0.1:9092');
$config->setBrokerVersion('1.0.0');
$config->setRequiredAck(1);
$config->setIsAsyn(false);
$config->setProduceInterval(500);
$producer = new \Kafka\Producer();

for($i = 0; $i < 100; $i++) {
    $producer->send([
        [
            'topic' => 'test',
            'value' => 'test... message #'.$i,
            'key' => '',
        ],
    ]);
}

6. Let’s send messages

Ok, we have all that we need. Start our consumer, run in a console.

php Consumer.php

Note that it can take some time before consumer begins to receive our messages.

Now try to send messages by running in a console in a new tab

php Producer.php

And

php ProducerSync.php

And as we can see all messages come to our consumer, exactly what we want!

What’s next?

We have a single-node instance with a single partition and one replica. Not many changes if we want to start a few more broker instances, just see example in official documentation.

 

 

https://github.com/weiboad/kafka-php

 

 

 

 

 

 

version: '2'

services:
  zookeeper:
    image: 'bitnami/zookeeper:3'
    ports:
      - '2181:2181'
    volumes:
      - 'zookeeper_data:/bitnami'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:2'
    ports:
      - '9092:9092'
    volumes:
      - 'kafka_data:/bitnami'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

Как добавить/удалить пользователя с помощью терминала в Ubuntu

Как добавить пользователя

Итак, чтобы добавить нового юзера в Ubuntu воспользуйтесь следующей командой:

Вместо “username” введите имя нового пользователя.

Создайте домашнюю папку для нового пользователя:

Теперь создайте пароль для нового пользователя:

Выставьте нужные права на домашнюю папку нового пользователя:

Всё, теперь вы можете пользоваться системой из под нового пользователя.

Как выдать root-права пользователю

Если вам нужно дать root-права новому юзеру, тогда воспользуйтесь данной командой:

Как удалить пользователя

Для удаления юзера воспользуйтесь двумя командами:

Так вы удалите самого пользователя из системы, а также его домашнюю папку.

Данная инструкция подходит к любой версии UbuntuLinux Mint, а также практически для любого другого дистрибутива.

How to install SSHFS

1 On MacOS
Using brew, install FUSE.

brew cask install osxfuse
Now install SSHFS:

brew install sshfs
2 On Ubuntu/Debian
sudo apt-get install sshfs

3 On Windows
Find the desired version of win-sshfs from this page and download it.

PHPUnit

wget https://phar.phpunit.de/phpunit.phar
chmod +x phpunit.phar
mv phpunit.phar /usr/local/bin/phpunit

git-ftp of Jenkins

Jenkins

git-ftp init -u $FTP_USERNAME -p $FTP_PASSWORD ftp://google.com/

ubuntu https://www.digitalocean.com/community/tutorials/how-to-install-jenkins-on-ubuntu-16-04

отдельно установить java

 

https://github.com/git-ftp/git-ftp/edit/master/INSTALL.md

 

# INSTALL

You can find instructions for:

- [Linux/Unix based systems using make](#linuxunix-based-systems-using-make)
- [Debian, Ubuntu and others using apt](#debian-ubuntu-and-others-using-apt)
- [ArchLinux](#archlinux-aur-unofficial)
- [MacOS](#macos)
- [Windows](#windows)


## Linux/Unix based systems using make

Note: Make sure Git and cURL are installed.

This should work on MacOS, Debian, Ubuntu, Fedora, RedHat, etc.

The easiest way is to use Git for installing:

```sh
git clone https://github.com/git-ftp/git-ftp.git
cd git-ftp

# choose the newest release
tag="$(git tag | grep '^[0-9]*\.[0-9]*\.[0-9]*$' | tail -1)"

# checkout the latest tag
git checkout "$tag"
sudo make install
```

Updating using git:

```sh
git fetch
git checkout "$(git tag | grep '^[0-9]*\.[0-9]*\.[0-9]*$' | tail -1)"
sudo make install
```


## Debian, Ubuntu and others using apt

At least Debian and Ubuntu provide git-ftp in their main repositories.

```sh
sudo apt-get install git-ftp
```

If you would like the newest release maintained by Git-ftp,
you can add the PPA:

```sh
sudo -s
add-apt-repository ppa:git-ftp/ppa

# On Debian, you need to modify the sources list to use the same PPA
source /etc/*-release
if [ "$ID" = "debian" ]; then
    dist="$(echo /etc/apt/sources.list.d/git-ftp-ppa-*.list | sed 's/^.*ppa-\(.*\)\.list$/\1/')"
    sed -i.backup "s/$dist/precise/g" /etc/apt/sources.list.d/git-ftp-ppa-*.list
fi

apt-get update
```


## ArchLinux (AUR: unofficial)

See https://aur.archlinux.org/packages/?O=0&C=0&SeB=nd&K=git-ftp&SB=v&SO=d&PP=50&do_Search=Go


## MacOS

First, ensure you have installed Xcode and command line tools. Command line tools can be download at https://developer.apple.com/download/more/ or via command: 

```
xcode-select --install
```

Using homebrew:

```sh
brew install git
brew install curl --with-libssh2
brew install brotli
brew install git-ftp
```

## Windows

There are at least two ways to install git-ftp on Windows.

- Using Git for Windows, former msysgit (recommended)
- Using cygwin

### Git for Windows, former msysgit (recommended)

Install [Git for Windows](https://git-for-windows.github.io/).
It comes with curl installed, but it doesn't support SFTP by default.
In order to use SFTP, [download curl](http://curl.haxx.se/download.html) for
Windows with SFTP support.
Win32 2000/XP MSI or Win64 2000/XP x86_64 MSI is recommended.
If you installed curl, then remove `bin/curl.exe` from your Git for Windows
installation directory. It will fall back to the newly installed version.

Finally, open the Git Bash which is located in `C:\Program Files (x86)\Git`
by default.

```bash
curl https://raw.githubusercontent.com/git-ftp/git-ftp/master/git-ftp > /bin/git-ftp
chmod 755 /bin/git-ftp
```

*Note: the `/bin` directory is an alias.
By default this is the same as `C:\Program Files (x86)\Git\usr\bin`.*

### cygwin

Install cygwin and install the package 'curl'.
Then open the cygwin console and install Git-ftp with the following commands:

```bash
curl https://raw.githubusercontent.com/git-ftp/git-ftp/master/git-ftp > /bin/git-ftp
chmod 755 /bin/git-ftp
```

### Git for Windows and cygwin both installed

If you have both Git for Windows and cygwin installed on Windows and want to
use Git for Windows for Git commands, you may get an error
"No such file or directory" for a path starting with "/cygdrive/", for example:

    creating `/cygdrive/c/TEMP/git-ftp-m7GH/delete_tmp': No such file or directory

The problem is that Git-ftp use commands from both Git for Windows and cygwin
directories. But by default, cygwin is configured to start paths with the
prefix "/cygdrive" while Git for Windows starts paths with "/".
To fix the problem, open file "<cygwin>\etc\fstab"
(e.g. "c:\cygwin\etc\fstab") and change parameter "/cygwin/" to "/", for example:

    # This is default:
    none /cygdrive/ cygdrive binary,posix=0,user 0 0

change to:

    # This is changed:
    none / cygdrive binary,posix=0,user 0 0

After this, close all console windows and try again.

 

vk.com/video13387403_456239249