utsubo – ページ 10

raspberry piでWifiを固定IPで使う

投稿者: utsubo 投稿日: 2016-03-19 in linux、raspberrypi

ハードウエア

raspberry pi B+

wifiアダプタ　WNG150U

OS

RASPBIAN JESSIE

設定

ハードウエアの確認

$ lsusb
Bus 001 Device 004: ID 04bb:094c I-O Data Device, Inc.

1 2	$ lsusb Bus 001 Device 004: ID 04bb:094c I-O Data Device, Inc.

ESSIDの確認

$ sudo iwlist wlan0 scan | grep ESSID
										ESSID:"ESSID"

1 2	$ sudo iwlist wlan0 scan \| grep ESSID ESSID:"ESSID"

wpa_supplicant.conf

$ sudo su 
#	chmod 660 root% chmod 660 /etc/wpa_suoplicant/wpa_supplicant.comf
# wpa_passphrase "SSID" "KEY" >> /etc/wpa_supplicant/wpa_supplicant.conf
# cat /etc/wpa_supplicant/wpa_supplicant.conf
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
	ssid="SSID"
	#psk="KEY"
	psk=ハッシュ化されたキー
}

$ sudo su

# chmod 660 root% chmod 660 /etc/wpa_suoplicant/wpa_supplicant.comf

# wpa_passphrase "SSID" "KEY" >> /etc/wpa_supplicant/wpa_supplicant.conf

# cat /etc/wpa_supplicant/wpa_supplicant.conf

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev

update_config=1

network={

ssid="SSID"

#psk="KEY"

psk=ハッシュ化されたキー

}

wpa_supplicant.confの編集

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
	ssid="SSID"
	proto=WPA2
	key_mgmt=WPA-PSK
	pairwise=TKIP CCMP
	group=TKIP CCMP
	#psk="KEY"
	psk=ハッシュ化されたキー
	scan_ssid=1
}
/etc/dhcpcd.conf
>||

interface wlan0
static ip_address=192.168.11.21/24
static routers=192.168.11.254
static domain_name_servers=192.168.10.1

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev

update_config=1

network={

ssid="SSID"

proto=WPA2

key_mgmt=WPA-PSK

pairwise=TKIP CCMP

group=TKIP CCMP

#psk="KEY"

psk=ハッシュ化されたキー

scan_ssid=1

}

/etc/dhcpcd.conf

>||

interface wlan0

static ip_address=192.168.11.21/24

static routers=192.168.11.254

static domain_name_servers=192.168.10.1

こんな感じでリブート

 $ ifconfig
eth0			Link encap:イーサネット	ハードウェアアドレス b8:27:eb:9c:30:07 
					inetアドレス:192.168.11.20 ブロードキャスト:192.168.11.255	マスク:255.255.255.0
					inet6アドレス: 240f:79:a8f8:1:b289:a029:fc2e:6ac/64 範囲:グローバル
					inet6アドレス: fe80::7474:4b62:ad78:2a00/64 範囲:リンク
					UP BROADCAST RUNNING MULTICAST	MTU:1500	メトリック:1
					RXパケット:2704 エラー:0 損失:594 オーバラン:0 フレーム:0
					TXパケット:607 エラー:0 損失:0 オーバラン:0 キャリア:0
			衝突(Collisions):0 TXキュー長:1000 
					RXバイト:161138 (157.3 KiB)	TXバイト:88612 (86.5 KiB)

lo				Link encap:ローカルループバック	
					inetアドレス:127.0.0.1 マスク:255.0.0.0
					inet6アドレス: ::1/128 範囲:ホスト
					UP LOOPBACK RUNNING	MTU:65536	メトリック:1
					RXパケット:140 エラー:0 損失:0 オーバラン:0 フレーム:0
					TXパケット:140 エラー:0 損失:0 オーバラン:0 キャリア:0
			衝突(Collisions):0 TXキュー長:0 
					RXバイト:11756 (11.4 KiB)	TXバイト:11756 (11.4 KiB)

wlan0		 Link encap:イーサネット	ハードウェアアドレス 34:76:c5:5d:7e:6c 
					inetアドレス:192.168.11.21 ブロードキャスト:192.168.11.255	マスク:255.255.255.0
					inet6アドレス: fe80::d437:7e2c:6380:2ccc/64 範囲:リンク
					inet6アドレス: 240f:79:a8f8:1:98a1:f100:4d2b:b96b/64 範囲:グローバル
					UP BROADCAST RUNNING MULTICAST	MTU:1500	メトリック:1
					RXパケット:3501 エラー:0 損失:632 オーバラン:0 フレーム:0
					TXパケット:119 エラー:0 損失:3 オーバラン:0 キャリア:0
			衝突(Collisions):0 TXキュー長:1000 
					RXバイト:582397 (568.7 KiB)	TXバイト:20034 (19.5 KiB)

$ ifconfig

eth0 Link encap:イーサネットハードウェアアドレス b8:27:eb:9c:30:07

inetアドレス:192.168.11.20 ブロードキャスト:192.168.11.255 マスク:255.255.255.0

inet6アドレス: 240f:79:a8f8:1:b289:a029:fc2e:6ac/64 範囲:グローバル

inet6アドレス: fe80::7474:4b62:ad78:2a00/64 範囲:リンク

UP BROADCAST RUNNING MULTICAST MTU:1500 メトリック:1

RXパケット:2704 エラー:0 損失:594 オーバラン:0 フレーム:0

TXパケット:607 エラー:0 損失:0 オーバラン:0 キャリア:0

衝突(Collisions):0 TXキュー長:1000

RXバイト:161138 (157.3 KiB) TXバイト:88612 (86.5 KiB)

lo Link encap:ローカルループバック

inetアドレス:127.0.0.1 マスク:255.0.0.0

inet6アドレス: ::1/128 範囲:ホスト

UP LOOPBACK RUNNING MTU:65536 メトリック:1

RXパケット:140 エラー:0 損失:0 オーバラン:0 フレーム:0

TXパケット:140 エラー:0 損失:0 オーバラン:0 キャリア:0

衝突(Collisions):0 TXキュー長:0

RXバイト:11756 (11.4 KiB) TXバイト:11756 (11.4 KiB)

wlan0 Link encap:イーサネットハードウェアアドレス 34:76:c5:5d:7e:6c

inetアドレス:192.168.11.21 ブロードキャスト:192.168.11.255 マスク:255.255.255.0

inet6アドレス: fe80::d437:7e2c:6380:2ccc/64 範囲:リンク

inet6アドレス: 240f:79:a8f8:1:98a1:f100:4d2b:b96b/64 範囲:グローバル

UP BROADCAST RUNNING MULTICAST MTU:1500 メトリック:1

RXパケット:3501 エラー:0 損失:632 オーバラン:0 フレーム:0

TXパケット:119 エラー:0 損失:3 オーバラン:0 キャリア:0

衝突(Collisions):0 TXキュー長:1000

RXバイト:582397 (568.7 KiB) TXバイト:20034 (19.5 KiB)

認識されました

RINEXデータから緯度経度を取得（簡易版）

投稿者: utsubo 投稿日: 2016-03-18 in GNSS

rinexデータのヘッダ部分にある「APPROX POSITION XYZ」から緯度経度を求めるスクリプト

rinexデータは国土地理院から取得できるファイルを使用します

lonlat.rb

#!/bin/env ruby 


class LonLat
	def exec(file)
		x,y,z=get_xyz(file)
		lat=Math.asin(z.to_f/6371000)*180.0/Math::PI
		lon=Math.atan2(y.to_f,x.to_f)*180.0/Math::PI
		p "latitude="+lat.to_s
		p "longitude="+lon.to_s
	end

	def get_xyz(file)
		open(file).each do |line|
			if line.include?("APPROX POSITION XYZ") then
				return line.gsub(/APPROX POSITION XYZ/,"").split(" ")
			end
		end
	end
end

if __FILE__ == $0 then
	if ARGV[0] == nil then
		p "usage:"+$0+" rinex.o"
		exit 0
	end
	LonLat.new.exec(ARGV[0])
end

#!/bin/env ruby

class LonLat

def exec(file)

x,y,z=get_xyz(file)

lat=Math.asin(z.to_f/6371000)*180.0/Math::PI

lon=Math.atan2(y.to_f,x.to_f)*180.0/Math::PI

p "latitude="+lat.to_s

p "longitude="+lon.to_s

end

def get_xyz(file)

open(file).each do |line|

if line.include?("APPROX POSITION XYZ") then

return line.gsub(/APPROX POSITION XYZ/,"").split(" ")

end

if __FILE__ == $0 then

if ARGV[0] == nil then

p "usage:"+$0+" rinex.o"

exit 0

end

LonLat.new.exec(ARGV[0])

end

実行結果

$ ruby lonlat.rb 00010700.11o
"latitude=45.1780484488787"
"longitude=141.7504468409626"

$ ruby lonlat.rb 00010700.11o

"latitude=45.1780484488787"

"longitude=141.7504468409626"

EC2インスタンスでTensorflow

投稿者: utsubo 投稿日: 2016-03-17 in AWS、ML

GoogleのTensorflow、GPUマシンでないとなかなか性能がでないので

EC2で作成してみます。

TensorflowはCUDA3.5以降対応だとかで、AWSのEC2インスタンスで使用可能なg2.2xlargeではCUDA3.0。ということでそのままでは使えないそうです

というわけで、いろいろ調べたところ、偉い方々が手順を示してくれています。

https://www.tecnos-dsm.co.jp/archives/info/technical_info_04

2016/3/17現在、これらの手順ですとTensorflowをコンパイルする際にエラーになります

$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
.......
ERROR: /home/ubuntu/tensorflow/WORKSPACE:16:6: First argument of load() is a path, not a label. It should start with a single slash if it is an absolute path..
ERROR: WORKSPACE file could not be parsed.
ERROR: no such package 'external': Package 'external' contains errors.
INFO: Elapsed time: 0.444s

$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

.......

ERROR: /home/ubuntu/tensorflow/WORKSPACE:16:6: First argument of load() is a path, not a label. It should start with a single slash if it is an absolute path..

ERROR: WORKSPACE file could not be parsed.

ERROR: no such package 'external': Package 'external' contains errors.

INFO: Elapsed time: 0.444s

これの回避策がこちらに

http://stackoverflow.com/questions/34941620/unable-to-build-tensorflow-from-source-with-bazel-22nd-january-2016

単純な話でbazelのバージョンのせいだとか。

bazelをコンパイルしなおします

git clone https://github.com/bazelbuild/bazel.git
cd bazel
git checkout tags/0.1.4
./compile.sh
sudo cp output/bazel /usr/bin

git clone https://github.com/bazelbuild/bazel.git

cd bazel

git checkout tags/0.1.4

./compile.sh

sudo cp output/bazel /usr/bin

その後、Tensorflowのコンパイル

$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
Extracting Bazel installation...
Sending SIGTERM to previous Bazel server (pid=11695)... done.
.......
INFO: Found 1 target...
INFO: From Executing genrule @png_archive//:configure [for host]:
/home/ubuntu/.cache/bazel/_bazel_ubuntu/ad1e09741bb4109fbc70ef8216b59ee2/tensorflow/external/png_archive/libpng-1.2.53 /home/ubuntu/.cache/bazel/_bazel_ubuntu/ad1e09741bb4109fbc70ef8216b59ee2/tensorflow
...

$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Extracting Bazel installation...

Sending SIGTERM to previous Bazel server (pid=11695)... done.

.......

INFO: Found 1 target...

INFO: From Executing genrule @png_archive//:configure [for host]:

/home/ubuntu/.cache/bazel/_bazel_ubuntu/ad1e09741bb4109fbc70ef8216b59ee2/tensorflow/external/png_archive/libpng-1.2.53 /home/ubuntu/.cache/bazel/_bazel_ubuntu/ad1e09741bb4109fbc70ef8216b59ee2/tensorflow

...

うまくいきました

Docker in DockerのJenkins環境をdockerで構築

投稿者: utsubo 投稿日: 2016-02-19 in docker

dockerで構築したjnekins内でdockerイメージを作成する

これができるとjenkinsでDockerイメージまで作成し、デプロイが簡単になります。

Dindイメージを使って作成してみます

Dockerfile

FROM docker:dind

############### add tools(jdk etc...) ####################
RUN apk update
RUN apk add wget git curl zip unzip	make bzip2 zlib openssl g++ libstdc++

# jdk
RUN apk add	openjdk7

# ant
RUN apk add apache-ant --update-cache --repository http://dl-4.alpinelinux.org/alpine/edge/testing/ --allow-untrusted
RUN echo "export PATH=$PATH:/usr/share/java/apache-ant/bin" >> /root/.bashrc

# node
RUN apk add	nodejs

# grunt
RUN npm install -g grunt-cli
# gulp
RUN npm install -g gulp

# Ruby
RUN apk add libffi libffi-dev gcc
RUN apk add ruby ruby-dev

# add
RUN apk add	expect

RUN apk add freetype freetype-dev fontconfig
RUN apk add xorg-server
RUN apk add bash
RUN apk add rsync --update-cache --repository http://dl-4.alpinelinux.org/alpine/edge/testing/ --allow-untrusted
RUN apk add openssh --update-cache --repository http://dl-4.alpinelinux.org/alpine/edge/testing/ --allow-untrusted

################## set up Jenkins ##########################

RUN apk --update add openjdk7 ttf-dejavu && rm -rf /var/cache/apk/*
ENV JENKINS_HOME /var/lib/jenkins

RUN mkdir -p /usr/local/jenkins
RUN adduser -D -H -s /bin/sh jenkins
RUN chown -R jenkins:jenkins /usr/local/jenkins/
ADD jenkins.war /usr/local/jenkins/jenkins.war
RUN chmod 644 /usr/local/jenkins/jenkins.war

# jst
RUN apk add tzdata --update-cache --repository http://dl-4.alpinelinux.org/alpine/edge/testing/ --allow-untrusted
RUN ln -sf	/usr/share/zoneinfo/Asia/Tokyo /etc/localtime

################### set process ##################
COPY jenkins.sh /usr/local/bin/jenkins.sh
ENTRYPOINT /usr/local/bin/start.sh && /bin/bash

FROM docker:dind

############### add tools(jdk etc...) ####################

RUN apk update

RUN apk add wget git curl zip unzip make bzip2 zlib openssl g++ libstdc++

# jdk

RUN apk add openjdk7

# ant

RUN apk add apache-ant --update-cache --repository http://dl-4.alpinelinux.org/alpine/edge/testing/ --allow-untrusted

RUN echo "export PATH=$PATH:/usr/share/java/apache-ant/bin" >> /root/.bashrc

# node

RUN apk add nodejs

# grunt

RUN npm install -g grunt-cli

# gulp

RUN npm install -g gulp

# Ruby

RUN apk add libffi libffi-dev gcc

RUN apk add ruby ruby-dev

# add

RUN apk add expect

RUN apk add freetype freetype-dev fontconfig

RUN apk add xorg-server

RUN apk add bash

RUN apk add rsync --update-cache --repository http://dl-4.alpinelinux.org/alpine/edge/testing/ --allow-untrusted

RUN apk add openssh --update-cache --repository http://dl-4.alpinelinux.org/alpine/edge/testing/ --allow-untrusted

################## set up Jenkins ##########################

RUN apk --update add openjdk7 ttf-dejavu && rm -rf /var/cache/apk/*

ENV JENKINS_HOME /var/lib/jenkins

RUN mkdir -p /usr/local/jenkins

RUN adduser -D -H -s /bin/sh jenkins

RUN chown -R jenkins:jenkins /usr/local/jenkins/

ADD jenkins.war /usr/local/jenkins/jenkins.war

RUN chmod 644 /usr/local/jenkins/jenkins.war

# jst

RUN apk add tzdata --update-cache --repository http://dl-4.alpinelinux.org/alpine/edge/testing/ --allow-untrusted

RUN ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime

################### set process ##################

COPY jenkins.sh /usr/local/bin/jenkins.sh

ENTRYPOINT /usr/local/bin/start.sh && /bin/bash

start.sh

#!/bin/sh
set -eo pipefail

# If there are any arguments then we want to run those instead
java -jar /usr/local/jenkins/jenkins.war --httpPort=8080	> /tmp/jenkins.log 2>&1 &	

set -e
docker daemon \
								--host=unix:///var/run/docker.sock \
								--host=tcp://0.0.0.0:2375 \
								--storage-driver=vfs \
--bip=172.17.101.1/24		# docker0のアドレス域を変更

#!/bin/sh

set -eo pipefail

# If there are any arguments then we want to run those instead

java -jar /usr/local/jenkins/jenkins.war --httpPort=8080 > /tmp/jenkins.log 2>&1 &

set -e

docker daemon \

--host=unix:///var/run/docker.sock \

--host=tcp://0.0.0.0:2375 \

--storage-driver=vfs \

--bip=172.17.101.1/24 # docker0のアドレス域を変更

docker-compose.xml

ついでにdocker-composeで起動するように設定します

myjenkins:
	restart: always
	image: myjenkins:1.0
	ports: 
		- "8080:8080"
	volumes:
		- /data/jenkins:/var/lib/jenkins		# host volumeをマップング
		- /data/git:/opt/git:ro
	env_file:
		- ./.docker-compose.env
	environment:
		- JAVA_OPTS=-XX:MaxPermSize=256m
	privileged: true

myjenkins:

restart: always

image: myjenkins:1.0

ports:

- "8080:8080"

volumes:

- /data/jenkins:/var/lib/jenkins # host volumeをマップング

- /data/git:/opt/git:ro

env_file:

- ./.docker-compose.env

environment:

- JAVA_OPTS=-XX:MaxPermSize=256m

privileged: true

.docker-compose.env

NO_PROXY=localhost,127.0.0.1
no_proxy=localhost,127.0.0.1
HTTP_PROXY=
HTTPS_PROXY=
http_proxy=
https_proxy=

NO_PROXY=localhost,127.0.0.1

no_proxy=localhost,127.0.0.1

HTTP_PROXY=

HTTPS_PROXY=

http_proxy=

https_proxy=

起動

# docker build -t myjenkins:1.0 .
# docker-compose -f docker-compose.yml up -d myjenkins

1 2	# docker build -t myjenkins:1.0 . # docker-compose -f docker-compose.yml up -d myjenkins

RでSHA

投稿者: utsubo 投稿日: 2016-02-16 in R

RでSHA1を作成する。serialize=Fを入れるのがポイント

&gt; install.packages("digest")
&gt; library(digest)
&gt; digest("abcdefg",algo="sha1",serialize=F)
[1] "2fb5e13419fc89246865e7a324f476ec624e8740"

> install.packages("digest")

> library(digest)

> digest("abcdefg",algo="sha1",serialize=F)

[1] "2fb5e13419fc89246865e7a324f476ec624e8740"

rubyの出力との比較

$ irb
irb(main):001:0&gt; require 'digest/sha1'
=&gt; true
irb(main):002:0&gt; Digest::SHA1.hexdigest("abcdefg")
=&gt; "2fb5e13419fc89246865e7a324f476ec624e8740"

$ irb

irb(main):001:0> require 'digest/sha1'

=> true

irb(main):002:0> Digest::SHA1.hexdigest("abcdefg")

=> "2fb5e13419fc89246865e7a324f476ec624e8740"

expectの書き方

投稿者: utsubo 投稿日: 2016-02-03 in linux

よく忘れるので備忘録

#!/bin/bash
my_ssh(){
/usr/bin/expect &lt;&lt; Eof
set timeout -1
spawn /usr/bin/ssh -o stricthostkeychecking=no root@hostname bash /path/to/script.sh
match_max 100000
expect -exact &quot;root@hostname&#039;s password: &quot;
send -- &quot;password\r&quot;
expect eof
Eof
}
my_ssh

#!/bin/bash

my_ssh(){

/usr/bin/expect << Eof

set timeout -1

spawn /usr/bin/ssh -o stricthostkeychecking=no root@hostname bash /path/to/script.sh

match_max 100000

expect -exact "root@hostname's password: "

send -- "password\r"

expect eof

Eof

}

my_ssh

EMRのSparkでWordCount

投稿者: utsubo 投稿日: 2016-01-27 in AWS

BODY:

EMRではSparkでファイルを開く際には*が使えるみたいだ　

package sample
import org.apache.spark._
import org.apache.spark.SparkContext._
import org.apache.spark.rdd.RDD.rddToPairRDDFunctions
object WordCount {
			def main(args: Array[String]) {
				println("wordcount,args="+args(0)+","+args(1))
			val conf = new SparkConf().setAppName("wordcount").setMaster("yarn-cluster")
			val sc = new SparkContext(conf)
			
				val textFile = sc.textFile(args(0))	// s3n://bucket/*gz
			val counts = textFile.flatMap(line => line.split(" "))
										 .map(word => (word, 1))
										 .reduceByKey(_ + _)
				println("counts="+counts.id+","+counts.name)
				println(counts.toDebugString)
			counts.saveAsTextFile(args(1))
			}
}

package sample

import org.apache.spark._

import org.apache.spark.SparkContext._

import org.apache.spark.rdd.RDD.rddToPairRDDFunctions

object WordCount {

def main(args: Array[String]) {

println("wordcount,args="+args(0)+","+args(1))

val conf = new SparkConf().setAppName("wordcount").setMaster("yarn-cluster")

val sc = new SparkContext(conf)

val textFile = sc.textFile(args(0)) // s3n://bucket/*gz

val counts = textFile.flatMap(line => line.split(" "))

.map(word => (word, 1))

.reduceByKey(_ + _)

println("counts="+counts.id+","+counts.name)

println(counts.toDebugString)

counts.saveAsTextFile(args(1))

}

こんな感じのBOWを数えるスクリプトを作成

s3にはgzで固められたファイルがたくさんある場合には

spark-submit --deploy-mode cluster --class sample.WordCount s3://bucket/dir/wordcount.jar s3n://bucket/log/*.gz s3n://bucket/output

1	spark-submit --deploy-mode cluster --class sample.WordCount s3://bucket/dir/wordcount.jar s3n://bucket/log/*.gz s3n://bucket/output

このような指定で起動すると全ファイルを解凍しながら計算し、outputへ結果を保存します

EMRでSparkSQLサンプル

投稿者: utsubo 投稿日: 2016-01-26 in AWS

SparkからHiveが使いづらいというか使えない？のでSparkSQLを使ってみました。

そこそこ試行錯誤する必要があったのでメモです。

データファイル

銘柄コード,日付,始値,高値,安値,終値,出来高

1	銘柄コード,日付,始値,高値,安値,終値,出来高

のフォーマットのファイルを用意しておきます。こんな感じ。

1301,2004-04-01,198,198,195,196,651000
1301,2004-04-02,194,196,194,196,490000
1301,2004-04-05,196,200,195,197,1478000
1301,2004-04-06,202,208,200,207,4324000

1301,2004-04-01,198,198,195,196,651000

1301,2004-04-02,194,196,194,196,490000

1301,2004-04-05,196,200,195,197,1478000

1301,2004-04-06,202,208,200,207,4324000

これをS3へアップしておきます

build.sbt

こんな感じで記述します。build assemblyでエラーが出るのでこんな記述にしています。

name := "spark_sample"

version := "1.0-SNAPSHOT"

scalaVersion := "2.11.7"

// additional libraries
libraryDependencies ++= Seq(
	"org.apache.spark" %% "spark-core" % "1.5.2" % "provided",
	"org.apache.spark" %% "spark-sql" % "1.5.2",
	"org.apache.spark" %% "spark-hive" % "1.5.2",
	"org.apache.spark" %% "spark-streaming" % "1.5.2",
	"org.apache.spark" %% "spark-streaming-kafka" % "1.5.2",
	"org.apache.spark" %% "spark-streaming-flume" % "1.5.2",
	"org.apache.spark" %% "spark-mllib" % "1.5.2",
	"org.apache.commons" % "commons-lang3" % "3.0",
	"org.eclipse.jetty"	% "jetty-client" % "8.1.14.v20131031",
	"com.typesafe.play" %% "play-json" % "2.3.10",
	"com.fasterxml.jackson.core" % "jackson-databind" % "2.6.4",
	"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.6.3",
	"org.elasticsearch" % "elasticsearch-hadoop-mr" % "2.0.0.RC1",
	"net.sf.opencsv" % "opencsv" % "2.0",
	"com.twitter.elephantbird" % "elephant-bird" % "4.5",
	"com.twitter.elephantbird" % "elephant-bird-core" % "4.5",
	"com.hadoop.gplcompression" % "hadoop-lzo" % "0.4.17",
	"mysql" % "mysql-connector-java" % "5.1.31",
	"com.datastax.spark" %% "spark-cassandra-connector" % "1.5.0-M3",
	"com.datastax.spark" %% "spark-cassandra-connector-java" % "1.5.0-M3",
	"com.github.scopt" %% "scopt" % "3.2.0",
	"org.scalatest" %% "scalatest" % "2.2.1" % "test",
	"com.holdenkarau" %% "spark-testing-base" %	"1.5.1_0.2.1",
	"org.apache.hive" % "hive-jdbc" % "1.2.1"
)

resolvers ++= Seq(
	"JBoss Repository" at "http://repository.jboss.org/nexus/content/repositories/releases/",
	"Spray Repository" at "http://repo.spray.cc/",
	"Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/",
	"Akka Repository" at "http://repo.akka.io/releases/",
	"Twitter4J Repository" at "http://twitter4j.org/maven2/",
	"Apache HBase" at "https://repository.apache.org/content/repositories/releases",
	"Twitter Maven Repo" at "http://maven.twttr.com/",
	"scala-tools" at "https://oss.sonatype.org/content/groups/scala-tools",
	"Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/",
	"Second Typesafe repo" at "http://repo.typesafe.com/typesafe/maven-releases/",
	"Mesosphere Public Repository" at "http://downloads.mesosphere.io/maven",
	Resolver.sonatypeRepo("public")
)

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
	{
		case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
		case m if m.startsWith("META-INF") => MergeStrategy.discard
		case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
		case PathList("org", "apache", xs @ _*) => MergeStrategy.first
		case PathList("org", "jboss", xs @ _*) => MergeStrategy.first
		case "about.html"	=> MergeStrategy.rename
		case "reference.conf" => MergeStrategy.concat
		case _ => MergeStrategy.first
	}
}

name := "spark_sample"

version := "1.0-SNAPSHOT"

scalaVersion := "2.11.7"

// additional libraries

libraryDependencies ++= Seq(

"org.apache.spark" %% "spark-core" % "1.5.2" % "provided",

"org.apache.spark" %% "spark-sql" % "1.5.2",

"org.apache.spark" %% "spark-hive" % "1.5.2",

"org.apache.spark" %% "spark-streaming" % "1.5.2",

"org.apache.spark" %% "spark-streaming-kafka" % "1.5.2",

"org.apache.spark" %% "spark-streaming-flume" % "1.5.2",

"org.apache.spark" %% "spark-mllib" % "1.5.2",

"org.apache.commons" % "commons-lang3" % "3.0",

"org.eclipse.jetty" % "jetty-client" % "8.1.14.v20131031",

"com.typesafe.play" %% "play-json" % "2.3.10",

"com.fasterxml.jackson.core" % "jackson-databind" % "2.6.4",

"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.6.3",

"org.elasticsearch" % "elasticsearch-hadoop-mr" % "2.0.0.RC1",

"net.sf.opencsv" % "opencsv" % "2.0",

"com.twitter.elephantbird" % "elephant-bird" % "4.5",

"com.twitter.elephantbird" % "elephant-bird-core" % "4.5",

"com.hadoop.gplcompression" % "hadoop-lzo" % "0.4.17",

"mysql" % "mysql-connector-java" % "5.1.31",

"com.datastax.spark" %% "spark-cassandra-connector" % "1.5.0-M3",

"com.datastax.spark" %% "spark-cassandra-connector-java" % "1.5.0-M3",

"com.github.scopt" %% "scopt" % "3.2.0",

"org.scalatest" %% "scalatest" % "2.2.1" % "test",

"com.holdenkarau" %% "spark-testing-base" % "1.5.1_0.2.1",

"org.apache.hive" % "hive-jdbc" % "1.2.1"

)

resolvers ++= Seq(

"JBoss Repository" at "http://repository.jboss.org/nexus/content/repositories/releases/",

"Spray Repository" at "http://repo.spray.cc/",

"Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/",

"Akka Repository" at "http://repo.akka.io/releases/",

"Twitter4J Repository" at "http://twitter4j.org/maven2/",

"Apache HBase" at "https://repository.apache.org/content/repositories/releases",

"Twitter Maven Repo" at "http://maven.twttr.com/",

"scala-tools" at "https://oss.sonatype.org/content/groups/scala-tools",

"Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/",

"Second Typesafe repo" at "http://repo.typesafe.com/typesafe/maven-releases/",

"Mesosphere Public Repository" at "http://downloads.mesosphere.io/maven",

Resolver.sonatypeRepo("public")

)

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>

{

case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard

case m if m.startsWith("META-INF") => MergeStrategy.discard

case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first

case PathList("org", "apache", xs @ _*) => MergeStrategy.first

case PathList("org", "jboss", xs @ _*) => MergeStrategy.first

case "about.html" => MergeStrategy.rename

case "reference.conf" => MergeStrategy.concat

case _ => MergeStrategy.first

}

ちなみにproject/assembly.sbtはこれ

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.1")

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.1")

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")

SqlSample.scala

http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-15-to-16

この辺りを参考に

package sample
import org.apache.spark.mllib.util.MLUtils
import org.apache.spark._
import org.apache.spark.api.java._
import org.apache.spark.sql._
import org.apache.spark.sql.types._

object SqlSample {
	def main(args: Array[String]) {
		val conf = new SparkConf().setAppName("SparkSQL").setMaster("yarn-cluster")
		val sc = new SparkContext(conf)	

		val sqlContext = new org.apache.spark.sql.SQLContext(sc)
		// Import Row.
		import org.apache.spark.sql.Row;

		// Import Spark SQL data types
		import org.apache.spark.sql.types.{StructType,StructField,StringType};

		val histRDD = sc.textFile(args(0)).map(_.split(",")).
			map(p => Row(p(0), p(1),p(2),p(3),p(4),p(5),p(6)))
		val schemaString = "code date open high low close volume"
		val schema =
				StructType(
				schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))	
				
		// Apply the schema to the RDD.
		val histDataFrame = sqlContext.createDataFrame(histRDD, schema)
		// Register the DataFrames as a table.
		histDataFrame.registerTempTable("priceHist")
		
		// SQL statements can be run by using the sql methods provided by sqlContext.
		val results = sqlContext.sql("SELECT code,date,open FROM priceHist where code='6758'")

		val ary=results.map(_.getValuesMap[Any](List("code", "date","open"))).collect()

		val outputLocation = args(1) // s3n://bucket/
		val data=sc.makeRDD(ary)
		data.saveAsTextFile(outputLocation)

		sc.stop()
	}
}

package sample

import org.apache.spark.mllib.util.MLUtils

import org.apache.spark._

import org.apache.spark.api.java._

import org.apache.spark.sql._

import org.apache.spark.sql.types._

object SqlSample {

def main(args: Array[String]) {

val conf = new SparkConf().setAppName("SparkSQL").setMaster("yarn-cluster")

val sc = new SparkContext(conf)

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

// Import Row.

import org.apache.spark.sql.Row;

// Import Spark SQL data types

import org.apache.spark.sql.types.{StructType,StructField,StringType};

val histRDD = sc.textFile(args(0)).map(_.split(",")).

map(p => Row(p(0), p(1),p(2),p(3),p(4),p(5),p(6)))

val schemaString = "code date open high low close volume"

val schema =

StructType(

schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))

// Apply the schema to the RDD.

val histDataFrame = sqlContext.createDataFrame(histRDD, schema)

// Register the DataFrames as a table.

histDataFrame.registerTempTable("priceHist")

// SQL statements can be run by using the sql methods provided by sqlContext.

val results = sqlContext.sql("SELECT code,date,open FROM priceHist where code='6758'")

val ary=results.map(_.getValuesMap[Any](List("code", "date","open"))).collect()

val outputLocation = args(1) // s3n://bucket/

val data=sc.makeRDD(ary)

data.saveAsTextFile(outputLocation)

sc.stop()

}

build

$ sbt package

1	$ sbt package

これで作成したJarを同じくS3へアップします

EMR

今までと同様にEMRを作成し、AddStepでSparkApplicationを追加します。Jarは先ほどアップしたものを指定します

Spark-submit options

--class sample.SqlSample

1	--class sample.SqlSample

Arguments

s3n://bucket/output

1	s3n://bucket/output

ここには出力ファイルが入ります

じっこすればOutputにMapで表現されたデータが保存されます

EMRでSparkサンプル

投稿者: utsubo 投稿日: 2016-01-25 in AWS

emr-4.2.0をベースにAdvancedOptionでSpark1.5.2を追加しクラスターを作成しておきます

今回はPiをモンテカルロシミュレーションで計算するSpark付属のサンプルプログラムをちょっと改造して使用します

build.sbt

build.sbtはこんな感じ

name := "spark_sample"

version := "1.0-SNAPSHOT"

scalaVersion := "2.11.7"

// additional libraries
libraryDependencies ++= Seq(
	"org.apache.spark" %% "spark-core" % "1.5.2",
	"org.apache.spark" %% "spark-sql" % "1.5.2",
	"org.apache.spark" %% "spark-mllib" % "1.5.2"
)

name := "spark_sample"

version := "1.0-SNAPSHOT"

scalaVersion := "2.11.7"

// additional libraries

libraryDependencies ++= Seq(

"org.apache.spark" %% "spark-core" % "1.5.2",

"org.apache.spark" %% "spark-sql" % "1.5.2",

"org.apache.spark" %% "spark-mllib" % "1.5.2"

)

SparkPi

SparkConfを作成する際のここがポイントです

		val conf = new SparkConf().setAppName("SparkPi").setMaster("yarn-cluster") // ここがポイント

1	val conf = new SparkConf().setAppName("SparkPi").setMaster("yarn-cluster") // ここがポイント

package sample

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.	See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.	You may obtain a copy of the License at
 *
 *		http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

import scala.math.random

import org.apache.spark.mllib.util.MLUtils
import org.apache.spark._

/** Computes an approximation to pi */
object SparkPi {
	def main(args: Array[String]) {
		val conf = new SparkConf().setAppName("SparkPi").setMaster("yarn-cluster") // ここがポイント
		val spark = new SparkContext(conf)
		val slices = 2
		val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
		val count = spark.parallelize(1 until n, slices).map { i =>
			val x = random * 2 - 1
			val y = random * 2 - 1
			if (x * x + y * y < 1) 1 else 0
		}.reduce(_ + _)
		println("Pi is roughly " + 4.0 * count / n)

		val outputLocation = args(0) // s3n://bucket/

		val pi = 4.0 * count / n
		val data = spark.makeRDD(Seq(pi))

		println(pi)
		data.saveAsTextFile(outputLocation)
		spark.stop()
	}
}

package sample

* Licensed to the Apache Software Foundation (ASF) under one or more

* contributor license agreements. See the NOTICE file distributed with

* this work for additional information regarding copyright ownership.

* The ASF licenses this file to You under the Apache License, Version 2.0

* (the "License"); you may not use this file except in compliance with

* the License. You may obtain a copy of the License at

* http://www.apache.org/licenses/LICENSE-2.0

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

import scala.math.random

import org.apache.spark.mllib.util.MLUtils

import org.apache.spark._

/** Computes an approximation to pi */

object SparkPi {

def main(args: Array[String]) {

val conf = new SparkConf().setAppName("SparkPi").setMaster("yarn-cluster") // ここがポイント

val spark = new SparkContext(conf)

val slices = 2

val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow

val count = spark.parallelize(1 until n, slices).map { i =>

val x = random * 2 - 1

val y = random * 2 - 1

if (x * x + y * y < 1) 1 else 0

}.reduce(_ + _)

println("Pi is roughly " + 4.0 * count / n)

val outputLocation = args(0) // s3n://bucket/

val pi = 4.0 * count / n

val data = spark.makeRDD(Seq(pi))

println(pi)

data.saveAsTextFile(outputLocation)

spark.stop()

}

ビルド

$ sbt packge

1	$ sbt packge

Jarファイルが作成されたらS3にアップしておきます

EMRでの実行

AWSコンソールからEMRで作成したクラスターを選択し、AddStepで先ほどアップしたJarファイルを指定し追加します

step typeにはSpark applicationを選択、

Spark－submit optionsに

--class sample.SparkPi --verbose

1	--class sample.SparkPi --verbose

Argumentsに出力を保存するS3のロケーションを入れておきます。すでにフォルダがあるとエラーになるので注意

s3n://bucketname/output

1	s3n://bucketname/output

実行後、出力先にファイルが作成されます

AmazonS3にjavaSDKを用いて文字列を書き込む

投稿者: utsubo 投稿日: 2016-01-13 in AWS、java

ポイントは２回InputStreamを作成することです。

http://stackoverflow.com/questions/8351886/amazons3-putobject-with-inputstream-length-example

package awssample;

import java.io.ByteArrayInputStream;

import java.io.InputStream;

import java.sql.SQLException;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.util.IOUtils;

public class S3Write {

	private static String endpoint = "https://s3-ap-northeast-1.amazonaws.com";

	public static void main(String[] args) throws SQLException {

		String s = "2011-01-01,9999,1,100,1.0,-1.0,0.5";
		// 認証オブジェクトを作成
		String accessKey = "xxxxxx";
		String accessSecretKey = "xxxxxxxx";
		AWSCredentials credentials = new BasicAWSCredentials(accessKey, accessSecretKey);

		// ConfigurationでTimeout時間を30秒に設定
		ClientConfiguration clientConfiguration = new ClientConfiguration();
		clientConfiguration.setConnectionTimeout(30000);

		// AmazonS3Clientをインスタンス化
		AmazonS3Client s3 = new AmazonS3Client(credentials, clientConfiguration);
		s3.setEndpoint(endpoint);
		try {
			InputStream is = new ByteArrayInputStream(s.getBytes("UTF-8"));
			byte[] contentBytes = IOUtils.toByteArray(is);
			Long contentLength = Long.valueOf(contentBytes.length);
			System.out.println("contentLength=" + contentLength + ",s=" + s);

			ObjectMetadata metadata = new ObjectMetadata();
			metadata.setContentLength(contentLength);
											 // isではなく new ByteStreamInputStreamでもう一度さくせいしたものを渡す
			s3.putObject(new PutObjectRequest("bucketname", "path/to/file.txt",
					new ByteArrayInputStream(s.getBytes("UTF-8")), metadata));

		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

package awssample;

import java.io.ByteArrayInputStream;

import java.io.InputStream;

import java.sql.SQLException;

import com.amazonaws.ClientConfiguration;

import com.amazonaws.auth.AWSCredentials;

import com.amazonaws.auth.BasicAWSCredentials;

import com.amazonaws.services.s3.AmazonS3Client;

import com.amazonaws.services.s3.model.ObjectMetadata;

import com.amazonaws.services.s3.model.PutObjectRequest;

import com.amazonaws.util.IOUtils;

public class S3Write {

private static String endpoint = "https://s3-ap-northeast-1.amazonaws.com";

public static void main(String[] args) throws SQLException {

String s = "2011-01-01,9999,1,100,1.0,-1.0,0.5";

// 認証オブジェクトを作成

String accessKey = "xxxxxx";

String accessSecretKey = "xxxxxxxx";

AWSCredentials credentials = new BasicAWSCredentials(accessKey, accessSecretKey);

// ConfigurationでTimeout時間を30秒に設定

ClientConfiguration clientConfiguration = new ClientConfiguration();

clientConfiguration.setConnectionTimeout(30000);

// AmazonS3Clientをインスタンス化

AmazonS3Client s3 = new AmazonS3Client(credentials, clientConfiguration);

s3.setEndpoint(endpoint);

try {

InputStream is = new ByteArrayInputStream(s.getBytes("UTF-8"));

byte[] contentBytes = IOUtils.toByteArray(is);

Long contentLength = Long.valueOf(contentBytes.length);

System.out.println("contentLength=" + contentLength + ",s=" + s);

ObjectMetadata metadata = new ObjectMetadata();

metadata.setContentLength(contentLength);

// isではなく new ByteStreamInputStreamでもう一度さくせいしたものを渡す

s3.putObject(new PutObjectRequest("bucketname", "path/to/file.txt",

new ByteArrayInputStream(s.getBytes("UTF-8")), metadata));

} catch (Exception e) {

e.printStackTrace();

}

Just another 靱技術士事務所 site

作者別: utsubo

raspberry piでWifiを固定IPで使う

ハードウエア

OS

設定

RINEXデータから緯度経度を取得（簡易版）

lonlat.rb

EC2インスタンスでTensorflow

Docker in DockerのJenkins環境をdockerで構築

Dockerfile

start.sh

docker-compose.xml

.docker-compose.env

起動

RでSHA

expectの書き方

EMRのSparkでWordCount

EMRでSparkSQLサンプル

データファイル

build.sbt

SqlSample.scala

build

EMR

Spark-submit options

Arguments

EMRでSparkサンプル

build.sbt

SparkPi

ビルド

EMRでの実行

AmazonS3にjavaSDKを用いて文字列を書き込む