tag:blogger.com,1999:blog-80961302024-03-14T11:23:51.345+03:30Dodo UnleashedBlogging from the Paradise Island of Mauritius Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.comBlogger378125tag:blogger.com,1999:blog-8096130.post-91179616785554676052018-04-25T23:11:00.002+04:302018-04-25T23:11:44.077+04:30Docker with Jenkins Plugins<div dir="ltr" style="text-align: left;" trbidi="on">
At times you might want to add have a version of Jenkins that automatically has a predefined set of plugins installed .<br />
<br />
You might want to find an existing jenkins with all the installed plugins assume the url is http://myjenkinshostname/jenkins to get the list of plugins you might need.<br />
<br />
Navigate to http://myjenkinshostname/jenkins<b>/script </b>there you want to past the following codes and execute :<br />
<br />
<span class="hljs-attribute" style="color: #50a14f; font-family: "Lucida Console", Monaco, monospace; font-size: 12.92px; white-space: pre;">def</span><span style="background-color: #fafafa; color: #383a42; font-family: "Lucida Console", Monaco, monospace; font-size: 12.92px; white-space: pre;"> plugins = jenkins.model.Jenkins.instance.getPluginManager().getPlugins()
plugins.each {</span><span class="hljs-attribute" style="color: #50a14f; font-family: "Lucida Console", Monaco, monospace; font-size: 12.92px; white-space: pre;">println</span><span style="background-color: #fafafa; color: #383a42; font-family: "Lucida Console", Monaco, monospace; font-size: 12.92px; white-space: pre;"> </span><span class="hljs-string" style="color: #50a14f; font-family: "Lucida Console", Monaco, monospace; font-size: 12.92px; white-space: pre;">"<span class="hljs-variable" style="color: #986801;">${it.getShortName()}</span>: <span class="hljs-variable" style="color: #986801;">${it.getVersion()}</span>"</span><span style="background-color: #fafafa; color: #383a42; font-family: "Lucida Console", Monaco, monospace; font-size: 12.92px; white-space: pre;">}</span><br />
<span style="background-color: #fafafa; color: #383a42; font-family: "Lucida Console", Monaco, monospace; font-size: 12.92px; white-space: pre;"><br /></span>
<br />
There are other ways to also get the list of plugins , <a href="http://www.noqcks.io/notes/2017/07/10/jenkins-installed-plugins-list/" target="_blank">check this site</a> .<br />
<br />
You would then create a docker file where you define all these necessary e.g<br />
<br />
===Dockerfile==================<br />
FROM jenkins/jenkins:lts<br />
<br />
ENV JENKINS_OPTS --prefix=/jenkins<br />
RUN /usr/local/bin/install-plugins.sh <span style="background-color: yellow;"> aws-credentials amazon-ecr amazon-ecs pipeline-maven</span><br />
<br />
==========================<br />
<br />
When the docker container is initialised this will automatically install all the plugins you define within it.<br />
<br />
Apparently there is a way to all install tools directly through a groovy script as mentioned on the official <a href="https://github.com/jenkinsci/docker/blob/master/README.md" target="_blank">github repo for jenkins</a> but I have yet to try it .<br />
<br />
This <a href="https://wiki.jenkins.io/display/JENKINS/Scriptler+Plugin" target="_blank">list of scripts </a>might then be interesting to try out for automatic installation of tools e.g to automatically<a href="https://wiki.jenkins.io/display/JENKINS/Add+a+Maven+Installation%2C+Tool+Installation%2C+Modify+System+Config" target="_blank"> install a specific version of maven</a> .<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-86843447331773060682018-03-17T22:29:00.000+03:302018-03-17T22:29:14.340+03:30Object Detection DL training with Tensorflow on GPU AWS<div dir="ltr" style="text-align: left;" trbidi="on">
Turns out that when if you want to train a model with say 5 types of different category of images you would need to make use of an Ec2 instance on AWS that has GPU capabilities.<br />
<br />
Else what happens with EC2 CPU instances is that they quickly run out of memory on the first dozen steps and the process gets killed .<br />
<br />
For that you would need at the very least a <b>p2.xlarge </b>and this is billed at around $0.9/hr ( at time of writing of this article) so still very expensive. So make sure that this VM is turned off the moment its not in use.<br />
<br />
I tried installing a vanilla p2.xlarge but ended up having issues with NVIDIA drivers so when you do launch an EC2 instance from AWS try to do so with an already configured AMI e.g AWS Deep Learning AMI.<br />
<br />
Follow the following steps:<br />
<ol style="text-align: left;">
<li>NVDIA drivers are properly installed - if your using the AWS Deep Learning AMI then chances are you don't need to worry about that</li>
<li>Then install <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/#os-requirements" target="_blank">Docker CE for Ubuntu</a></li>
<li>Ensure that the following <a href="https://docs.docker.com/install/linux/linux-postinstall/" target="_blank">post-installation instructions </a> are also covered</li>
<li>Then install <a href="https://github.com/NVIDIA/nvidia-docker" target="_blank">NVIDIA Docker</a> using instructions on the page.</li>
</ol>
<div>
<br /></div>
<div>
You should then be able to launch your docker instance using the following command:</div>
<div>
<br /></div>
<br />
<div>
<span style="background-color: #f3f3f3;">docker volume create notebooks</span></div>
<div>
<div>
<span style="background-color: #f3f3f3;"><br /></span></div>
<div>
<span style="background-color: #f3f3f3;">nvidia-docker run -it --name tensorflow -v notebooks:/notebooks -p 8888:8888 -p 6006:6006 gcr.io/tensorflow/tensorflow:latest-gpu</span></div>
</div>
<br />
<div>
<br /></div>
<div>
Once you got the container running then its just a question of following my other articles to continue with the training:<br />
<br />
1. <a href="https://javedmandary.blogspot.com/2018/02/tensorflow-running-in-docker-deployed.html" target="_blank">Setup of environment , in my case using Docker</a><br />
2. <a href="https://javedmandary.blogspot.com/2018/03/object-detection-labelling-image.html" target="_blank">Labeling and creation of tfRecord</a><br />
3. <a href="https://javedmandary.blogspot.com/2018/03/training-custom-object.html" target="_blank">Training Custom Object Detection</a></div>
<div>
<br /></div>
<div>
So typically you would use the GPU instances to train your models and CPU instances only to run test against your frozen inference graph for example using jupyter as less expensive.</div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-31150183374898978982018-03-12T16:23:00.002+03:302018-03-12T16:32:19.172+03:30Mauritius Heat Map for Real Estate Prices per m2 using Python<div dir="ltr" style="text-align: left;" trbidi="on">
Yesterday I thought I would give a try to see whether it is possible to create a heatmap of real estate prices per m2 in Mauritius , this covers the prices of apartments , houses and villas.<br />
<br />
Given that there is no dataset to work against readily I managed to collect information from popular online sources that had prices of real estate in an attempt to then plot them on our Mauritian map.<br />
<br />
I used only Python and Jupyter notebook for all the process of collecting data , analysing and plotting.<br />
<br />
The process gathering data off from web pages is called <a href="https://www.dataquest.io/blog/web-scraping-beautifulsoup/" target="_blank">webscraping </a>and I will not give out codes in regards to this if you are not careful enough you might cause DoS attacks against the websites and you don't want that.<br />
<br />
I used <a href="https://www.crummy.com/software/BeautifulSoup/" target="_blank">beautifulsoup4 </a> to parse through the retrieved HTML data to retrieve specifically what data I was interested in to create some <a href="https://pandas.pydata.org/" target="_blank">pandas </a>dataframes .<br />
<br />
The first analysis based on the data showed the following :<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-Rkd6TZKnEDo/WqZuBagdxyI/AAAAAAABjQU/iIPY7ZeZbV0b0tGemOThOhcmvOZhxQ5igCLcBGAs/s1600/area_price.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="393" data-original-width="404" height="193" src="https://4.bp.blogspot.com/-Rkd6TZKnEDo/WqZuBagdxyI/AAAAAAABjQU/iIPY7ZeZbV0b0tGemOThOhcmvOZhxQ5igCLcBGAs/s200/area_price.JPG" width="200" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
The price in Rs per m2 is lower in the center than on the North and its much more expensive that on the East of the island . However note that 4000+ records were used on which an average was made for each Area.<br />
<br />
Below are the regions where the prices are the lowest per m2 in Mauritius :<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-VK88vwD_ryg/WqZu5Lq59uI/AAAAAAABjQc/cF6HGTUeyxw7USOYOxsZBfTeB8oRxEJCgCLcBGAs/s1600/lowest_regions.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="651" data-original-width="675" height="308" src="https://3.bp.blogspot.com/-VK88vwD_ryg/WqZu5Lq59uI/AAAAAAABjQc/cF6HGTUeyxw7USOYOxsZBfTeB8oRxEJCgCLcBGAs/s320/lowest_regions.JPG" width="320" /></a></div>
<br />
And here are the regions which have the highest prices m2 in Mauritius :<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-yhl7H9O5HRU/WqZvi56ayUI/AAAAAAABjQo/dANByof1uSU4hwkRKTNE6MFuUtNDjhE_ACLcBGAs/s1600/highest_regions.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1499" data-original-width="684" height="640" src="https://2.bp.blogspot.com/-yhl7H9O5HRU/WqZvi56ayUI/AAAAAAABjQo/dANByof1uSU4hwkRKTNE6MFuUtNDjhE_ACLcBGAs/s640/highest_regions.JPG" width="291" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Note that there are over 150 regions so I cannot show the full list but that gives a good enough indication of prices where the costal regions obviously being more expensive.<br />
<br />
Then I used the Google Maps API to get the coordinates based on the region name , the code for this can be seen below, function name is geodataMapper , it also handles the problem that if you are making a number of subsequent calls to Google Maps API (in this case over 150+) , then at one point you will get an exception saying something like "too many retries" :<br />
<script src="https://gist.github.com/anonymous/3687eca2d0a5085826fe5883558fc01c.js"></script>
<br />
<br />
To retrieve latitude and longitude data for all the regions i through the following code that will then iterate through a panda dataframe containing the region and price m2 and for each region call the above geodataMapper , at the end it will create a csv file containing region , price per m2 , latitude and longitude :<br />
<br />
<script src="https://gist.github.com/anonymous/ae04886191a32651ed533b0cda3b43a9.js"></script>
<br />
Now once you have this data you can install <a href="https://github.com/pbugnion/gmaps" target="_blank">gmaps</a> which is awesome for creating some basic heatmaps and it even has extension for jupyter notebook .<br />
<br />
Below are the codes for it , all you need is a pandas dataframe from the csv you saved:<br />
<br />
<script src="https://gist.github.com/anonymous/3ee10eed7c74b639b4339363aa45cbdd.js"></script>
<br />
And you should be able to see something like this appearing on your jupyter notebook:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-jg19myB_WUI/WqZ2dRsUjvI/AAAAAAABjQ4/9vOa3k8hBGchA-jopPGHjiz1zxUnNOikwCLcBGAs/s1600/MauritiusSmall.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="334" data-original-width="317" height="320" src="https://4.bp.blogspot.com/-jg19myB_WUI/WqZ2dRsUjvI/AAAAAAABjQ4/9vOa3k8hBGchA-jopPGHjiz1zxUnNOikwCLcBGAs/s320/MauritiusSmall.JPG" width="303" /></a></div>
<br />
Which is pretty neat I still need to work out how to create the heat map based on google map boundaries instead of just one actual point but that I will leave for another time.<br />
<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-77849339296202615152018-03-04T17:50:00.000+03:302018-03-04T17:51:12.547+03:30Training Custom Object<div dir="ltr" style="text-align: left;" trbidi="on">
The following activities have been done:<br />
<br />
1. <a href="https://javedmandary.blogspot.com/2018/02/tensorflow-running-in-docker-deployed.html" target="_blank">Setup of environment , in my case using Docker</a><br />
2. <a href="https://javedmandary.blogspot.com/2018/03/object-detection-labelling-image.html" target="_blank">Labeling and creation of tfRecord</a><br />
<br />
Now we need to launch the actual training of tensorflow on the custom object . I have been following the tutorial from <a href="https://pythonprogramming.net/training-custom-objects-tensorflow-object-detection-api-tutorial/?completed=/creating-tfrecord-files-tensorflow-object-detection-api-tutorial/" target="_blank">python programming</a> to do that.<br />
<br />
<b>Docker </b><br />
<br />
1. Start by downloading a copy of<b> ssd_mobilenet_v1_coco_11_06_2017.tar.gz</b>:<br />
<br />
<span style="background-color: #f3f3f3;">wget http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_11_06_2017.tar.gz</span><br />
<br />
2. Copy this to your object_detection folder:<br />
<br />
<span style="background-color: #f3f3f3;">docker cp ssd_mobilenet_v1_coco_11_06_2017.tar.gz tensorflow:/notebooks/models/research/object_detection/</span><br />
<br />
3. Then untar the model:<br />
<span style="background-color: #f3f3f3;">tar -xvzf ssd_mobilenet_v1_coco_11_06_2017.tar.gz</span><br />
<br />
4. Now taking the <b>ssd_mobilenet_v1_pets.config </b> copy it to :<br />
<span style="background-color: #f3f3f3;">docker cp ssd_mobilenet_v1_pets.config tensorflow:/notebooks/models/research/object_detection/training</span><br />
<br />
======<b>ssd_mobilenet_v1_pets.config</b> =====<br />
# SSD with Mobilenet v1, configured for the mac-n-cheese dataset.<br />
# Users should configure the fine_tune_checkpoint field in the train config as<br />
# well as the label_map_path and input_path fields in the train_input_reader and<br />
# eval_input_reader. Search for "${YOUR_GCS_BUCKET}" to find the fields that<br />
# should be configured.<br />
<br />
model {<br />
ssd {<br />
num_classes: 1<br />
box_coder {<br />
faster_rcnn_box_coder {<br />
y_scale: 10.0<br />
x_scale: 10.0<br />
height_scale: 5.0<br />
width_scale: 5.0<br />
}<br />
}<br />
matcher {<br />
argmax_matcher {<br />
matched_threshold: 0.5<br />
unmatched_threshold: 0.5<br />
ignore_thresholds: false<br />
negatives_lower_than_unmatched: true<br />
force_match_for_each_row: true<br />
}<br />
}<br />
similarity_calculator {<br />
iou_similarity {<br />
}<br />
}<br />
anchor_generator {<br />
ssd_anchor_generator {<br />
num_layers: 6<br />
min_scale: 0.2<br />
max_scale: 0.95<br />
aspect_ratios: 1.0<br />
aspect_ratios: 2.0<br />
aspect_ratios: 0.5<br />
aspect_ratios: 3.0<br />
aspect_ratios: 0.3333<br />
}<br />
}<br />
image_resizer {<br />
fixed_shape_resizer {<br />
height: 300<br />
width: 300<br />
}<br />
}<br />
box_predictor {<br />
convolutional_box_predictor {<br />
min_depth: 0<br />
max_depth: 0<br />
num_layers_before_predictor: 0<br />
use_dropout: false<br />
dropout_keep_probability: 0.8<br />
kernel_size: 1<br />
box_code_size: 4<br />
apply_sigmoid_to_scores: false<br />
conv_hyperparams {<br />
activation: RELU_6,<br />
regularizer {<br />
l2_regularizer {<br />
weight: 0.00004<br />
}<br />
}<br />
initializer {<br />
truncated_normal_initializer {<br />
stddev: 0.03<br />
mean: 0.0<br />
}<br />
}<br />
batch_norm {<br />
train: true,<br />
scale: true,<br />
center: true,<br />
decay: 0.9997,<br />
epsilon: 0.001,<br />
}<br />
}<br />
}<br />
}<br />
feature_extractor {<br />
type: 'ssd_mobilenet_v1'<br />
min_depth: 16<br />
depth_multiplier: 1.0<br />
conv_hyperparams {<br />
activation: RELU_6,<br />
regularizer {<br />
l2_regularizer {<br />
weight: 0.00004<br />
}<br />
}<br />
initializer {<br />
truncated_normal_initializer {<br />
stddev: 0.03<br />
mean: 0.0<br />
}<br />
}<br />
batch_norm {<br />
train: true,<br />
scale: true,<br />
center: true,<br />
decay: 0.9997,<br />
epsilon: 0.001,<br />
}<br />
}<br />
}<br />
loss {<br />
classification_loss {<br />
weighted_sigmoid {<br />
anchorwise_output: true<br />
}<br />
}<br />
localization_loss {<br />
weighted_smooth_l1 {<br />
anchorwise_output: true<br />
}<br />
}<br />
hard_example_miner {<br />
num_hard_examples: 3000<br />
iou_threshold: 0.99<br />
loss_type: CLASSIFICATION<br />
max_negatives_per_positive: 3<br />
min_negatives_per_image: 0<br />
}<br />
classification_weight: 1.0<br />
localization_weight: 1.0<br />
}<br />
normalize_loss_by_num_matches: true<br />
post_processing {<br />
batch_non_max_suppression {<br />
score_threshold: 1e-8<br />
iou_threshold: 0.6<br />
max_detections_per_class: 100<br />
max_total_detections: 100<br />
}<br />
score_converter: SIGMOID<br />
}<br />
}<br />
}<br />
<br />
train_config: {<br />
batch_size:<b> 10</b><br />
optimizer {<br />
rms_prop_optimizer: {<br />
learning_rate: {<br />
exponential_decay_learning_rate {<br />
initial_learning_rate: 0.004<br />
decay_steps: 800720<br />
decay_factor: 0.95<br />
}<br />
}<br />
momentum_optimizer_value: 0.9<br />
decay: 0.9<br />
epsilon: 1.0<br />
}<br />
}<br />
fine_tune_checkpoint: "<b>ssd_mobilenet_v1_coco_11_06_2017/</b>model.ckpt"<br />
from_detection_checkpoint: true<br />
data_augmentation_options {<br />
random_horizontal_flip {<br />
}<br />
}<br />
data_augmentation_options {<br />
ssd_random_crop {<br />
}<br />
}<br />
}<br />
<br />
train_input_reader: {<br />
tf_record_input_reader {<br />
input_path: "data/train.record"<br />
}<br />
label_map_path: "data/object-detection.pbtxt"<br />
}<br />
<br />
eval_config: {<br />
num_examples: 40<br />
}<br />
<br />
eval_input_reader: {<br />
tf_record_input_reader {<br />
input_path: "data/test.record"<br />
}<br />
label_map_path: "training/object-detection.pbtxt"<br />
shuffle: false<br />
num_readers: 1<br />
}<br />
====================================<br />
<br />
5. Copy the following <b>object-detection.pbtxt</b> to the object_detection/data directory:<br />
<br />
<span style="background-color: #eeeeee;">docker cp object-detection.pbtxt tensorflow:/notebooks/models/research/<b>object_detection/data</b></span><br />
<br />
========object-detection.pbtxt============<br />
<br />
item {<br />
id: 1<br />
name: '<span style="color: red;">object_label_name</span>'<br />
}<br />
<br />
====================================<br />
<br />
In my case am identifying only one object so there is only 1 item , change <span style="color: red;">object_label_name </span>to the name of your label you defined when annotating your images .<br />
<br />
6. <b>Launch the training</b> using the following command:<br />
<br />
<span style="background-color: #eeeeee;">python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config</span><br />
<br />
<b>Note </b>if there are any errors try doing the following before executing train.py:<br />
<span style="background-color: #eeeeee;">cd /notebooks/models/research/</span><br />
<span style="background-color: #eeeeee;">export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim</span><br />
<span style="background-color: #eeeeee;">cd object_detection</span><br />
<br />
<br />
You should start seeing steps being executed as below :<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-K3oIJ5dCcDc/Wpv-nN-LzPI/AAAAAAABjJ0/rSQN9QFuB7Y2nGKLoGc3bTPPl7IUW_T8ACLcBGAs/s1600/Learning_in_progress.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="837" data-original-width="1600" height="209" src="https://2.bp.blogspot.com/-K3oIJ5dCcDc/Wpv-nN-LzPI/AAAAAAABjJ0/rSQN9QFuB7Y2nGKLoGc3bTPPl7IUW_T8ACLcBGAs/s400/Learning_in_progress.JPG" width="400" /></a></div>
<br />
<br />
<b>Tensorboard</b><br />
<br />
To monitor progress on tensorboard use the following command , can be in another docker exec putty window:<br />
<br />
<span style="background-color: #eeeeee; color: rgba(0 , 0 , 0 , 0.87); font-family: monospace , monospace; font-size: 15px;">tensorboard --logdir='training'</span><br />
<br />
<br />
You should be able to see the board on http://your_DOMAIN_OR_IP:<b>6006</b>/ , this took some secs for me before it actually was shown :<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-ZEQeQGUF0vA/WpwAKm1MCsI/AAAAAAABjKA/KsvWSonnKqob_PmFvzNYiiqm1X0p5a0uQCLcBGAs/s1600/tensorboard.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1022" data-original-width="1600" height="255" src="https://3.bp.blogspot.com/-ZEQeQGUF0vA/WpwAKm1MCsI/AAAAAAABjKA/KsvWSonnKqob_PmFvzNYiiqm1X0p5a0uQCLcBGAs/s400/tensorboard.JPG" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<br />
<br />
<br />
<div>
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-73238305772518184582018-03-04T14:42:00.000+03:302018-03-04T17:50:59.138+03:30Object Detection Labelling image and generating tfRecord<div dir="ltr" style="text-align: left;" trbidi="on">
I made use of the tutorial from <a href="http://www.jackyle.com/2017/12/train-customize-object-for-object.html" target="_blank">jackyle </a> to label my images . Note that <a href="https://pythonprogramming.net/" target="_blank">pythonprogramming </a>has also the exact <a href="https://pythonprogramming.net/creating-tfrecord-files-tensorflow-object-detection-api-tutorial/" target="_blank">same tutoria</a>l :) !<br />
<br />
Mind you the hardest part is really finding the images , the rest goes more or less pretty fast.<br />
<br />
Basically you use the tool <a href="https://github.com/tzutalin/labelImg" target="_blank">labelImage </a>to help in the labelling , which basically creates an XML file for each of the image that you label .<br />
<br />
I used the windows binary which can be found <a href="https://tzutalin.github.io/labelImg/" target="_blank">here </a>and did all the labelling from windows itself.<br />
<br />
Your directory structure should be like this under ROOT_DIR/models/research/object_detection:<br />
<br />
|-<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">xml_to_csv.py</span><br />
|-data<br />
|-images<br />
|- train<br />
|- test<br />
<br />
<br />
Once you have labelled all your images you need to do the following :<br />
<br />
1. Place 70 % of your images + xml in a folder images/train<br />
2. Place 30% of your images + xml in a folder images/test<br />
3. Create a <span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">xml_to_csv.py file that looks like below:</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">==========</span><span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><b>xml_to_csv.py</b>====================</span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">import os</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">import glob</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">import pandas as pd</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">import xml.etree.ElementTree as ET</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">def xml_to_csv(path):</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> xml_list = []</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> for xml_file in glob.glob(path + '/*.xml'):</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> tree = ET.parse(xml_file)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> root = tree.getroot()</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> for member in root.findall('object'):</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> value = (root.find('filename').text,</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> int(root.find('size')[0].text),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> int(root.find('size')[1].text),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> member[0].text,</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> int(member[4][0].text),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> int(member[4][1].text),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> int(member[4][2].text),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> int(member[4][3].text)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> )</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> xml_list.append(value)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax']</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> xml_df = pd.DataFrame(xml_list, columns=column_name)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> return xml_df</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">def main():</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> for directory in ['<b>train</b>','<b>test</b>']:</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> image_path = os.path.join(os.getcwd(), 'images/{}'.format(directory))</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> xml_df = xml_to_csv(image_path)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> xml_df.to_csv('<b>data</b>/{}_labels.csv'.format(directory), index=None)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> print('Successfully converted xml to csv.')</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: white; font-size: 15px;"></span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">main()</span></span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">========================================</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">4. Excecute </span><span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">python xml_to_csv.py , this will read all the xml files and create 2 csv files in the <b>data </b>directory train_labels.csv and test_labels.csv</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><b>Docker Container</b></span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">If you installed tensorflow using docker container ( <a href="https://javedmandary.blogspot.com/2018/02/tensorflow-running-in-docker-deployed.html" target="_blank">check my tutorial</a> ) and cloned the following repository ( install git if you dont already have it ):</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="background-color: white; color: blue; font-family: monospace; font-size: 15px;">git clone https://github.com/tensorflow/models.git </span><br />
<span style="background-color: white; color: blue; font-family: monospace; font-size: 15px;"><br /></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">You can copy a zip of the images folder , images.zip , and the python </span></span><span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">xml_to_csv.py into the container, tensorflow, using :</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: #eeeeee; font-size: 15px;">docker cp xml_to_csv.py tensorflow:/notebooks/models/research/object_detection/</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: #f3f3f3; font-size: 15px;">docker cp images.zip tensorflow:/notebooks/models/research/object_detection/</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">Now all you need to do is to unzip the images ( install unzip if you dont already have it) :</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: #eeeeee; font-size: 15px;">unzip images.zip</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">Then you connect to the running instance of the container using :</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: #eeeeee; font-size: 15px;">docker exec -it tensorflow /bin/bash</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">and execute :</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: white; font-size: 15px;"></span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: #eeeeee; font-size: 15px;">python xml_to_csv.py</span></span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: white; font-size: 15px;"><b>Generating TfRecord</b></span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: white; font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: white; font-size: 15px;">Now the next step is based on the generated test_labels.csv and train_labels.csv we are going to create tensorflow record files for each .</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: white; font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: white; font-size: 15px;">1. Copy the following </span><span style="font-size: 15px;">generate_tfrecord.py</span><span style="background-color: white; font-size: 15px;"> file into your /</span></span><span style="background-color: #eeeeee; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">notebooks/models/research/object_detection/ </span><span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"> directory:</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">=========</span><span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><b>generate_tfrecord.py</b>=========================================</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">"""</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">Usage:</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> # From tensorflow/models/</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> # Create train data:</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record <b>--images_path=images/train</b></span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> # Create test data:</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=data/test.record <b>--images_path=images/test</b></span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">"""</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">from __future__ import division</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">from __future__ import print_function</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">from __future__ import absolute_import</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">import os</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">import io</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">import pandas as pd</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">import tensorflow as tf</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">from PIL import Image</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">from object_detection.utils import dataset_util</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">from collections import namedtuple, OrderedDict</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">flags = tf.app.flags</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">flags.DEFINE_string('csv_input', '', 'Path to the CSV input')</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">flags.DEFINE_string('output_path', '', 'Path to output TFRecord')</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><b>flags.DEFINE_string('images_path', '', 'Path to Images')</b></span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">FLAGS = flags.FLAGS</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"># TO-DO replace this with label map</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">def class_text_to_int(row_label):</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> if row_label == 'cocacola':</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> return 1</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> else:</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> None</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">def split(df, group):</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> data = namedtuple('data', ['filename', 'object'])</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> gb = df.groupby(group)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">def create_tf_example(group, path):</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid:</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> encoded_jpg = fid.read()</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> encoded_jpg_io = io.BytesIO(encoded_jpg)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> image = Image.open(encoded_jpg_io)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> width, height = image.size</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> filename = group.filename.encode('utf8')</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> image_format = b'jpg'</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> xmins = []</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> xmaxs = []</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> ymins = []</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> ymaxs = []</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> classes_text = []</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> classes = []</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> for index, row in group.object.iterrows():</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> xmins.append(row['xmin'] / width)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> xmaxs.append(row['xmax'] / width)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> ymins.append(row['ymin'] / height)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> ymaxs.append(row['ymax'] / height)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> classes_text.append(row['class'].encode('utf8'))</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> classes.append(class_text_to_int(row['class']))</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> tf_example = tf.train.Example(features=tf.train.Features(feature={</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/height': dataset_util.int64_feature(height),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/width': dataset_util.int64_feature(width),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/filename': dataset_util.bytes_feature(filename),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/source_id': dataset_util.bytes_feature(filename),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/encoded': dataset_util.bytes_feature(encoded_jpg),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/format': dataset_util.bytes_feature(image_format),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/object/class/text': dataset_util.bytes_list_feature(classes_text),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> 'image/object/class/label': dataset_util.int64_list_feature(classes),</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> }))</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> return tf_example</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">def main(_):</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> writer = tf.python_io.TFRecordWriter(FLAGS.output_path)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> path = os.path.join(os.getcwd(), <b>FLAGS.images_path</b>)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> examples = pd.read_csv(FLAGS.csv_input)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> grouped = split(examples, 'filename')</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> for group in grouped:</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> tf_example = create_tf_example(group, path)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> writer.write(tf_example.SerializeToString())</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> writer.close()</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> output_path = os.path.join(os.getcwd(), FLAGS.output_path)</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> print('Successfully created the TFRecords: {}'.format(output_path))</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">if __name__ == '__main__':</span></span><br />
<br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"> tf.app.run()</span></span><br />
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="font-size: 15px;">===================================================================</span></span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: white; font-size: 15px;"><b>Note </b>that its the same file that is mentioned in the </span></span><a href="http://www.jackyle.com/2017/12/train-customize-object-for-object.html" target="_blank">jackyle </a> tutorial however I kept getting file not found exceptions as it was trying to get the image from the images directory directly instead of images/test or images/train. So I made some modifications such as the images directory for train and test could be passed as a flag.<br />
<br />
2. Execute for following command to make sure Python is on your path:<br />
<br />
<span style="background-color: #f3f3f3;">cd /notebooks/models/research/</span><br />
<span style="background-color: #f3f3f3;">export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim</span><br />
<span style="background-color: #f3f3f3;">cd object_detection</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">3. Then create the train record:</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --images_path=images/train</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">4. Create the test record :</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=data/test.record --images_path=images/test</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;">You should now have 2 files train.record and test.record under the </span><span style="color: #333333; font-family: "arial" , "geneva" , sans-serif;"><span style="background-color: white; font-size: 15px;">/</span></span><span style="color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><span style="background-color: #eeeeee;">notebooks/models/research/object_detection/</span><span style="background-color: #eeeeee; font-weight: bold;">data </span><span style="background-color: white;"><b> </b>directory.</span></span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "geneva" , sans-serif; font-size: 15px;"><br /></span></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-57700337608808349722018-02-26T15:36:00.001+03:302018-02-26T15:36:13.078+03:30Object Detection Tensorflow<div dir="ltr" style="text-align: left;" trbidi="on">
To get started with object detection have a look at the following jupyter notebook:<div>
<br /></div>
<div>
<a href="https://github.com/tensorflow/models/blob/master/research/object_detection/object_detection_tutorial.ipynb">https://github.com/tensorflow/models/blob/master/research/object_detection/object_detection_tutorial.ipynb</a> </div>
<div>
<br /></div>
<div>
Assuming that you have already setup your environment with tensorflow , in my case its a docker container . You need to still execute the following <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md" target="_blank">instructions</a> .</div>
<div>
<br /></div>
<div>
One issue i was getting was that the jupyter notebook kept failing at the following line despite having followed all the instructions:</div>
<div>
<br /></div>
<div>
<span style="background-color: orange;">from <b>object_detection.utils</b> import ops as utils_ops</span></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
So i discovered that this was due to Python libraries not being availabe in PYTHONPATH:</div>
<div>
<br /></div>
<div>
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span> From tensorflow/models/research/</span>
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">export</span> PYTHONPATH=<span class="pl-smi" style="box-sizing: border-box;">$PYTHONPATH</span>:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">`</span>pwd<span class="pl-pds" style="box-sizing: border-box;">`</span></span>:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">`</span>pwd<span class="pl-pds" style="box-sizing: border-box;">`</span></span>/slim</pre>
</div>
<div>
<br /></div>
<div>
if despite having executed the above in your container or your tensorflow environment the problem still persists in your Jupyter notebook consider adding it directly as can be seen below :</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
====Extract Jupyter Notebook=============================</div>
<div>
<div>
import numpy as np</div>
<div>
import os</div>
<div>
import six.moves.urllib as urllib</div>
<div>
import sys</div>
<div>
import tarfile</div>
<div>
import tensorflow as tf</div>
<div>
import zipfile</div>
<div>
from timeit import default_timer as timer</div>
<div>
import cv2</div>
<div>
<br /></div>
<div>
<span style="background-color: cyan;">sys.path.append('/notebooks/models/research') # point to your tensorflow dir</span></div>
<div>
<span style="background-color: cyan;">sys.path.append('/notebooks/models/research/slim') # point ot your slim dir</span></div>
<div>
<br /></div>
<div>
from collections import defaultdict</div>
<div>
from io import StringIO</div>
<div>
from matplotlib import pyplot as plt</div>
<div>
from PIL import Image</div>
<div>
from object_detection.utils import ops as utils_ops</div>
<div>
<br /></div>
<div>
if tf.__version__ < '1.4.0':</div>
<div>
raise ImportError('Please upgrade your tensorflow installation to v1.4.* or later!')</div>
</div>
<div>
<br /></div>
<div>
============================================================</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Note that I have also changed the default method to use opencv for faster image IO and a timer to determine performance.</div>
<div>
<br /></div>
<div>
You install opencv using :</div>
<div>
<pre style="background-color: #eeeeee; border: 1px solid rgb(170, 170, 170); color: #333333; line-height: 30px; margin-bottom: 30px; padding: 10px; white-space: pre-wrap;">sudo apt-get install python-opencv</pre>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
==============Extract ========================</div>
<div>
<div>
for image_path in TEST_IMAGE_PATHS:</div>
<div>
<span style="background-color: lime;"> start = timer()</span></div>
<div>
<br /></div>
<div>
#image = Image.open(image_path)</div>
<div>
<span style="background-color: lime;"> image = cv2.imread(image_path)</span></div>
<div>
</div>
<div>
# the array based representation of the image will be used later in order to prepare the</div>
<div>
# result image with boxes and labels on it.</div>
<div>
<br /></div>
<div>
#image_np = load_image_into_numpy_array(image)</div>
<div>
<span style="background-color: lime;"> image_np = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)</span></div>
<div>
</div>
<div>
# Expand dimensions since the model expects images to have shape: [1, None, None, 3]</div>
<div>
#image_np_expanded = np.expand_dims(image_np, axis=0)</div>
<div>
<span style="background-color: lime;"> image_np_expanded = np.expand_dims(image_np, axis=0)</span></div>
<div>
<br /></div>
<div>
# Actual detection.</div>
<div>
output_dict = run_inference_for_single_image(image_np, detection_graph)</div>
<div>
# Visualization of the results of a detection.</div>
<div>
vis_util.visualize_boxes_and_labels_on_image_array(</div>
<div>
image_np,</div>
<div>
output_dict['detection_boxes'],</div>
<div>
output_dict['detection_classes'],</div>
<div>
output_dict['detection_scores'],</div>
<div>
category_index,</div>
<div>
instance_masks=output_dict.get('detection_masks'),</div>
<div>
use_normalized_coordinates=True,</div>
<div>
line_thickness=8)</div>
<div>
plt.figure(figsize=IMAGE_SIZE)</div>
<div>
plt.imshow(image_np)</div>
<div>
<span style="background-color: lime;"> end = timer()</span></div>
<div>
<span style="background-color: lime;"> duration = (end - start)</span></div>
<div>
<span style="background-color: lime;"><br /></span></div>
<div>
<span style="background-color: lime;"> print('Image: {0} took {1} to be processed'.format(image_path,duration))</span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
===========================================</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-71028745298390059712018-02-20T23:12:00.000+03:302018-02-20T23:12:00.699+03:30Installing an SSL certificate for NGinx on Ubuntu <div dir="ltr" style="text-align: left;" trbidi="on">
Never thought that it would be that easy with a tool called <a href="https://letsencrypt.org/" target="_blank">Letsencrypt</a> but basically if you want to add SSL certificate on your Nginx server all you have to do is follow the instructions <a href="https://certbot.eff.org/#ubuntuxenial-nginx" target="_blank">here</a> that is :<br />
<br />
<br />
<br />
- Update / Install the following packages:<br />
<pre style="background-color: #f3f3f3; border-radius: 0px; color: #111111; font-family: VeraMono; font-size: 14px; margin-bottom: 2em; margin-top: 2em; overflow: auto; padding: 0px; word-wrap: break-word;">$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install python-certbot-nginx </pre>
then you ask <a href="https://certbot.eff.org/" target="_blank">certbot</a> to install :<br />
<pre style="background-color: #f3f3f3; border-radius: 0px; color: #111111; font-family: VeraMono; font-size: 14px; margin-bottom: 2em; margin-top: 2em; overflow: auto; padding: 0px; word-wrap: break-word;">sudo certbot --nginx</pre>
<b>Note</b>: You will be asked for a domain name , ip addresses will not be allowed<br />
<br />
Also certificates need to be renewed , cerbot can take care of that according to documentation</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-87969645988415762202018-02-16T11:16:00.001+03:302018-03-17T22:37:06.921+03:30TensorFlow running in Docker deployed on Ubuntu<div dir="ltr" style="text-align: left;" trbidi="on">
This entry will provide a view of the different steps required to setup <a href="https://www.tensorflow.org/" target="_blank">TensorFlow </a>on a Ubuntu environment by running it within a docker container .<br />
<br />
Whether your starting off on Machine learning with <a href="https://www.tensorflow.org/" target="_blank">TensorFlow </a>or your a veteran and you want to setup an infra with a docker container running Tensorflow then this article is for you<br />
<br />
<br />
<b>Update (17/3/2018): </b>For doing the setup using GPU instances on AWS check <a href="https://javedmandary.blogspot.com/2018/03/object-detection-dl-training-with.html" target="_blank">my article here</a> <br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-Pi6rnjaeE9k/WoavLSk619I/AAAAAAABjB0/2Eu04WMR8eoqIL75ijW11vUSeE8prQKLACLcBGAs/s1600/docker_jupyter-1-e1455998494759.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="339" data-original-width="500" height="216" src="https://3.bp.blogspot.com/-Pi6rnjaeE9k/WoavLSk619I/AAAAAAABjB0/2Eu04WMR8eoqIL75ijW11vUSeE8prQKLACLcBGAs/s320/docker_jupyter-1-e1455998494759.png" width="320" /></a></div>
<br />
<br />
<b>Why we want to use docker ?</b><br />
<br />
<ul style="text-align: left;">
<li>We do not want to have our configurations for Tensorflow being messed up by other python versions and configs for other applications so we are isolating it</li>
<li>Installing from Docker image is very practical and saves us a lot of time so we can focus directly on the our coding </li>
</ul>
<br />
<br />
<b>Prerequisites </b>:<br />
<br />
1. Ubuntu VM , am using one with 8Gb RAM , 100Gb SSD<br />
2. Ubuntu CE installed , follow t<a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/" target="_blank">his link </a><br />
3. Make sure you have TCP access to 8888, 9000 if running on AWS ( or other cloud platforms )<br />
<br />
Once docker has been installed do make sure that non-root users can also execute the docker command by following instructions from <a href="https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user" target="_blank">docker site</a> .<br />
<br />
<b>Running Tensorflow Container</b><br />
<b><br /></b>
At time of writing the current version of tensorflow is 1.5 so kick start just execute the following command which is document <a href="https://www.tensorflow.org/versions/r0.12/get_started/os_setup#docker_installation" target="_blank">here </a>.<br />
<br />
<br />
<span class="pln" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">docker run </span><span class="pun" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">-</span><span class="pln" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">it </span><span class="pun" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">-</span><span class="pln" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">p </span><span class="lit" style="background-color: #f7f7f7; box-sizing: inherit; color: #c53929; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">8888</span><span class="pun" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">:</span><span class="lit" style="background-color: #f7f7f7; box-sizing: inherit; color: #c53929; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">8888</span><span class="pln" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;"> gcr</span><span class="pun" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">.</span><span class="pln" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">io</span><span class="pun" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">/</span><span class="pln" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">tensorflow</span><span class="pun" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">/</span><span class="pln" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;">tensorflow</span><br />
<div>
<span class="pln" style="background-color: #f7f7f7; box-sizing: inherit; color: #37474f; font-family: "roboto mono" , monospace; font-size: 14px; white-space: pre;"><br /></span></div>
<div>
<br /></div>
<div>
Note that there will be some output when the docker container runs with a token url please copy paste and keep somewhere .<br />
<br />
<b>Using Tensorflow</b><br />
<br />
When launching the run command for the TensorFlow docker container above a url with a token looking something like this would be shown on your console:<br />
http://your_ip_adddress:8888/?token=eXXXXXXXXXXXXXXXXXXXXXXXXX<br />
<br />
Use this url directly to login to your jupyter notebook.<br />
<br />
Now this is great but we need to also create a volume such as we can easily access files for example pulled from a git repository .<br />
<br />
So you might want to remove the docker container that you just started in the last step and use docker-compose file below<br />
<br />
<b>Docker compose</b><br />
<br />
Docker compose needs to be installed using the following <a href="https://docs.docker.com/compose/install/#install-compose" target="_blank">instructions </a>.<br />
<br />
Here is a simple docker-compose file to be able to run the command to run a Tensorflow :<br />
<br />
------------docker-compose-tensorflow.yml--------------------------------------<br />
<br />
version: '3'<br />
services:<br />
tensorflow:<br />
build: .<br />
image: gcr.io/tensorflow/tensorflow<br />
container_name: tensorflow<br />
volumes:<br />
- notebooks:/notebooks<br />
ports:<br />
- "8888:8888"<br />
- "6006:6006"<br />
environment:<br />
- IMAGE_SIZE=224<br />
- ARCHITECTURE=mobilenet_0.50_224<br />
<br />
volumes:<br />
notebooks:<br />
<br />
<br />
-----------------------------------------------------------------------------------------<br />
<br />
The command then to start the container is simple<br />
<br />
<span style="background-color: #eeeeee;">docker-compose -f docker-compose-tensorflow <b>up </b></span><br />
<br />
and to stop the container :<br />
<br />
<span style="background-color: #eeeeee;">docker-compose -f docker-compose-tensorflow <b>down</b></span><br />
<br /></div>
<div>
Note that the docker-compose file contains a port mapping for 6006 which is used for <a href="https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard" target="_blank">Tensorboard </a>and a volume mapping to <b>notebooks .</b><br />
<br />
The volume notebooks ensure that you persist your notebooks on subsequent up - down cycles of your container . Else you would lose all your contents on each shutdown of the container.<br />
<br />
<br /></div>
<b>Managing containers </b><br />
<br />
The best way I have found to manage by containers in a practical manner is through Portainer and you can install this on docker using the following command ( <a href="https://portainer.io/install.html" target="_blank">site here</a>):<br />
<br />
<span style="background-color: #f3f3f3;">$docker volume create portainer_data</span><br />
<span style="background-color: #f3f3f3;">$docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer</span><br />
<br />
Now login to http://your_ip_adddress:9000/ to set a password for admin<br />
<br />
<br />
<b>Container shell access </b><br />
<br />
If you want to connect to your running container's shell and assuming that this is called tensorflow as in our case just do :<br />
<span style="background-color: #f3f3f3;"><br /></span>
<span style="background-color: #f3f3f3;">docker exec -it tensorflow /bin/bash </span><br />
<br />
Else you can also use Portainer as explained below .<br />
<br />
<b>Accessing Tensorflow</b><br />
<br />
You should be able to log onto your docker containers directly through Portainer by clicking on the container name and clicking on console and the click connect on bash<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-DujFIho7OiM/WofkcowduNI/AAAAAAABjCk/m3hX96XYOlcwKXFTjDswcblwYlX13ofWQCLcBGAs/s1600/portainer_console.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="703" data-original-width="1600" height="140" src="https://4.bp.blogspot.com/-DujFIho7OiM/WofkcowduNI/AAAAAAABjCk/m3hX96XYOlcwKXFTjDswcblwYlX13ofWQCLcBGAs/s320/portainer_console.JPG" width="320" /></a></div>
<br />
<br />
This by default will give you access to <b>/notebook </b>directory :<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-VRjPLKlLzds/WoflDIX-xjI/AAAAAAABjCs/bmCKL12b6So5Fjd22XCph8OFtng5cIwGgCLcBGAs/s1600/bash.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="666" data-original-width="1416" height="150" src="https://2.bp.blogspot.com/-VRjPLKlLzds/WoflDIX-xjI/AAAAAAABjCs/bmCKL12b6So5Fjd22XCph8OFtng5cIwGgCLcBGAs/s320/bash.JPG" width="320" /></a></div>
<br />
<br />
I found it particularly useful to use this feature of portainer as it meant that directly from portainer web app you could access the bash of your running container .<br />
<br />
Also keeping files under /notebook allows you to view them through your jupyter notebook instance.<br />
<br />
<br />
<br />
<b>Reverse Proxy (optional step)</b><br />
<br />
Although not absolutely required I find it useful to be able to access to the tools all directly from port 80 you can install a reverse proxy in front of the Portainer and Tensorflow Jupyter notebook by installing nginx .<br />
<br />
Installation is pretty straight forward please check instructions <a href="https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-16-04" target="_blank">here </a> :<br />
<br />
<ul class="prefixed" style="background-color: rgba(0, 0, 0, 0.05); box-sizing: border-box; color: #3a3a3a; font-family: monospace; font-size: 14px; margin: 0px; padding: 0px;">
<li class="line" prefix="$" style="box-sizing: border-box; line-height: 1.5 !important; list-style: disc outside none; margin-bottom: 0.8rem; margin-left: 0px; margin-right: 0px !important; margin-top: 0px !important; padding-bottom: 0px !important; padding-left: 4px; padding-right: 0px !important; padding-top: 0px !important; white-space: pre;">sudo apt-get update
</li>
<li class="line" prefix="$" style="box-sizing: border-box; line-height: 1.5 !important; list-style: disc outside none; margin-bottom: 0.8rem; margin-left: 0px; margin-right: 0px !important; margin-top: 0px !important; padding-bottom: 0px !important; padding-left: 4px; padding-right: 0px !important; padding-top: 0px !important; white-space: pre;">sudo apt-get install nginx</li>
</ul>
<br />
Now assuming you are using the default ports as mentioned above (else modify as required ) you need to create a file with a *.conf e.g myreveseproxysettings.conf (or whatever suits you) , then you <b>sudo cp</b> (copy) this file to directory <b>/etc/nginx/conf.d </b><br />
<b><br /></b>
Not that nginx main config includes configurations files which have a *.conf extension within the /etc/nginx/conf.d directory .<br />
<br />
Remember to change <span style="color: red;">My_IP_ADDRESS_OR_DOMAIN_NAME</span> with your IP Address or Domain name .<br />
<br />
==================myreveseproxysettings.conf=============================<br />
<br />
<br />
server {<br />
listen 80;<br />
listen [::]:80;<br />
server_name <span style="color: red;">My_IP_ADDRESS_OR_DOMAIN_NAME</span>;<br />
# root /usr/share/nginx/html;<br />
<br />
proxy_set_header Host $http_host;<br />
proxy_set_header X-Real-IP $remote_addr;<br />
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />
<br />
<span style="white-space: pre;"> </span><br />
<span style="white-space: pre;"> </span>location <span style="background-color: yellow;">/portainer/</span> {<br />
proxy_http_version 1.1;<br />
<br />
proxy_set_header Upgrade $http_upgrade;<br />
proxy_set_header Connection "upgrade";<br />
<br />
proxy_pass "http://localhost:<span style="background-color: yellow;">9000</span>/"; <br />
}<br />
<span style="white-space: pre;"> </span><br />
location<span style="background-color: lime;"> /</span> {<br />
proxy_pass "http://localhost:<span style="background-color: lime;">8888</span>/";<br />
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />
<span style="white-space: pre;"> </span> proxy_set_header X-Real-IP $remote_addr;<br />
<span style="white-space: pre;"> </span> proxy_set_header Host $http_host;<br />
<span style="white-space: pre;"> </span> proxy_http_version 1.1;<br />
<span style="white-space: pre;"> </span> proxy_redirect off;<br />
<span style="white-space: pre;"> </span> proxy_buffering off;<br />
<span style="white-space: pre;"> </span> proxy_set_header Upgrade $http_upgrade;<br />
<span style="white-space: pre;"> </span> proxy_set_header Connection "upgrade";<br />
<span style="white-space: pre;"> </span> proxy_read_timeout 86400;<br />
}<br />
<br />
error_page 404 /404.html;<br />
location = /40x.html {<br />
}<br />
<br />
error_page 500 502 503 504 /50x.html;<br />
location = /50x.html {<br />
}<br />
}<br />
<br />
<br />
=================================================<br />
<br />
Following commands are useful:<br />
<br />
1. Start nginx<br />
<br />
<span style="background-color: rgba(0 , 0 , 0 , 0.05); color: #3a3a3a; font-family: monospace; font-size: 14px; white-space: pre;">sudo systemctl start nginx</span><br />
<span style="background-color: rgba(0 , 0 , 0 , 0.05); color: #3a3a3a; font-family: monospace; font-size: 14px; white-space: pre;"><br /></span>
2. Stop nginx<br />
<br />
<span style="background-color: rgba(0 , 0 , 0 , 0.05); color: #3a3a3a; font-family: monospace; font-size: 14px; white-space: pre;">sudo systemctl stop nginx</span><br />
<span style="background-color: rgba(0 , 0 , 0 , 0.05); color: #3a3a3a; font-family: monospace; font-size: 14px; white-space: pre;"><br /></span>
3. Check status nginx<br />
<br />
<span style="background-color: rgba(0 , 0 , 0 , 0.05); color: #3a3a3a; font-family: monospace; font-size: 14px; white-space: pre;">systemctl status nginx</span><br />
<br />
<br />
To get out of the status message just do a " :" followed by a "q"<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-34227977112225238942018-01-03T16:24:00.000+03:302018-01-03T16:24:17.301+03:30Generation of project using JHipster-UML<div dir="ltr" style="text-align: left;" trbidi="on">
Just found out that now this command needs to be used to generate using JHipster-UML:<br />
<span style="background-color: rgba(27, 31, 35, 0.05); color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.9px;">yo jhipster:import-jdl yourUMLFile.jh</span><br />
<span style="background-color: rgba(27, 31, 35, 0.05); color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.9px;"><br /></span>
where <span style="background-color: rgba(27, 31, 35, 0.05); color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.9px;">yourUMLFile.jh </span> is the file containing the JHipster UML definitions.<br />
<br />
Once genrated then start the app using mvnw</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-30420155752518027992017-11-23T05:35:00.002+03:302017-11-23T05:35:36.403+03:30Environment setup scikit-learn on Windows<div dir="ltr" style="text-align: left;" trbidi="on">
Currently starting to tinker with <a href="http://scikit-learn.org/stable/" target="_blank">scikit-learn</a> for Machine learning , i found it a bit confusing to know where to start from a Windows perspective given I didn't have much knowledge around python .<br />
<br />
So what you should be doing to get started setting up your environment ( at least whats working for me ) is to install Anaconda 3.x whilst choosing the 64 or 32 bit depending on your environment:<br />
<br />
<a href="https://www.anaconda.com/download/">https://www.anaconda.com/download/</a><br />
<br />
The installation is pretty much straight forward from there .<br />
<br />
You will also need to have installed GIT:<br />
<br />
<a href="https://git-scm.com/download/win">https://git-scm.com/download/win</a><br />
<br />
Open up the Anaconda prompt and execute the command to install scikit<br />
<br />
<span style="background-color: #eeeff0; color: #333333; font-family: Consolas, "Liberation Mono", Courier, monospace; font-size: 16px;">conda install -c anaconda scikit-learn </span><br />
<br />
refer to : <a href="https://anaconda.org/anaconda/scikit-learn">https://anaconda.org/anaconda/scikit-learn</a></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-35458230285376888012017-02-23T11:52:00.000+03:302017-02-23T11:53:34.681+03:30Amazon Lex Speech Permissions<div dir="ltr" style="text-align: left;" trbidi="on">
If you are planning to include speech recognition features in your Amazon Lex enabled chatbot you should add a specific <b>policy </b>to the role against which you are executing your command .<br />
<div>
<br /></div>
<div>
Basically you need to give rights to Amazon Polly to your specific role .</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-qJUCy9Rlvmk/WK6bi54Kk7I/AAAAAAAApN8/4eSc7xh45BgR25Cnq-4moD8IlRIWa5rPACLcB/s1600/create_role_policy.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://4.bp.blogspot.com/-qJUCy9Rlvmk/WK6bi54Kk7I/AAAAAAAApN8/4eSc7xh45BgR25Cnq-4moD8IlRIWa5rPACLcB/s400/create_role_policy.PNG" width="222" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
The screenshot below shows what you need to add.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-k3O5UqVZnGc/WK6amUEX5_I/AAAAAAAApN0/ARu0iKyl0aUMjy4kwA0MB7PuXY5AGhsCwCLcB/s1600/policy_polly.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="296" src="https://1.bp.blogspot.com/-k3O5UqVZnGc/WK6amUEX5_I/AAAAAAAApN0/ARu0iKyl0aUMjy4kwA0MB7PuXY5AGhsCwCLcB/s400/policy_polly.PNG" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Content:</div>
<div>
<br /></div>
<div>
<div>
{</div>
<div>
"Version": "2012-10-17",</div>
<div>
"Statement": [</div>
<div>
{</div>
<div>
"Sid": "AllowAllPollyActions",</div>
<div>
"Effect": "Allow",</div>
<div>
"Action": [</div>
<div>
"polly:*"</div>
<div>
],</div>
<div>
"Resource": "*"</div>
<div>
}</div>
<div>
]</div>
<div>
}</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-125932107975950522017-01-11T21:23:00.000+03:302017-01-11T21:23:13.827+03:30ElasticSearch , Logstash , Kibana and Filebeat with Docker<div dir="ltr" style="text-align: left;" trbidi="on">
When you have a number of containers in your DevOps infrastructure running you might need at some point in time to monitor the logs from your container managed apps .<br />
<br />
One solution which works ( at least for me ) is by using <a href="https://www.elastic.co/products/elasticsearch" target="_blank">ElasticSearch </a>, <a href="https://www.elastic.co/products/logstash" target="_blank">Logstash </a>, <a href="https://www.elastic.co/products/kibana" target="_blank">Kibana </a>or also called <a href="https://www.elastic.co/" target="_blank">ELK </a> , to capture and parse your logs whilst having a tool like <a href="https://www.elastic.co/products/beats/filebeat" target="_blank">Filebeat </a>which actually monitors your logs from <a href="https://www.docker.com/" target="_blank">Docker </a>containers ( or not ) and send updates across to the ELK server.<br />
<br />
I have created a github repository with my solution using ELK + Filebeat and Docker , have a look at the guide around how to setup :<br />
<a href="https://github.com/javedgit/Docker-ELK">https://github.com/javedgit/Docker-ELK</a></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-47344071036383944072017-01-09T15:21:00.001+03:302017-01-09T15:21:23.948+03:30Install docker in 2 commands on Ubuntu<div dir="ltr" style="text-align: left;" trbidi="on">
Simpliest way I found to install docker on ubuntu :<br />
<br />
1. <span style="background-color: black; color: white;"><code style="border-radius: 3px; border: 0px; font-size: 14px; margin: 0px; padding: 0px; text-align: justify; white-space: pre-wrap;">wget -qO- https://get.docker.com/ | sh</code><span style="font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; text-align: justify;"> </span></span><br />
<span style="background-color: white; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; text-align: justify;">2. </span><span style="background-color: black; font-family: monospace; font-size: 14px; text-align: justify; white-space: pre-wrap;"><span style="color: white;">sudo usermod -aG docker $(whoami)</span></span><br />
<span style="background-color: white; font-family: monospace; font-size: 14px; text-align: justify; white-space: pre-wrap;"><br /></span>
<span style="background-color: white; font-family: monospace; font-size: 14px; text-align: justify; white-space: pre-wrap;">Then logout and login back to your terminal .</span><br />
<br />
<div style="text-align: justify;">
<span style="font-family: monospace;"><span style="background-color: white; font-size: 14px; white-space: pre-wrap;">Execute </span><span style="background-color: black; font-size: 14px; white-space: pre-wrap;"><span style="color: white;">docker ps</span></span><span style="background-color: white; font-size: 14px; white-space: pre-wrap;"> to see if docker is installed correctly.</span></span></div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-62705208088108666092017-01-06T18:13:00.003+03:302017-01-06T18:14:59.682+03:30Automatic Install of Maven with Jenkins and use within Pipeline<div dir="ltr" style="text-align: left;" trbidi="on">
Assume you want a specific version of Maven to be installed automatically when doing a build e.g because you need to have a build executed on a remote node.<br />
<br />
This is what you need to do to perform this :<br />
<br />
<br />
<ul style="text-align: left;">
<li>Install the following 2 plugins on your Jenkins 2x instance ( if not already preseent :</li>
<ul>
<li>Config File Provider -<a href="https://wiki.jenkins-ci.org/display/JENKINS/Config+File+Provider+Plugin" target="_blank"> https://wiki.jenkins-ci.org/display/JENKINS/Config+File+Provider+Plugin</a> </li>
<li>Pipeline Maven Plugin - <a href="https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Maven+Plugin">https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Maven+Plugin</a> </li>
</ul>
<li>Restart your Jenkins such as the Plugins can take effect</li>
<li>Define your settings.xml file within <i>Jenkins > Manage Jenkins > Managed Files</i></li>
<ul>
<li>Click on <i>Add a new Config</i></li>
<li>choose <i>Global Maven settings </i></li>
<li>choose a valid id name e.g GlobalMavenSettingsForXXX</li>
<li>copy paste the settings.xml of your project on next page</li>
<ul>
<li>Note the settings.xml should allow you to get your plugins and dependencies from your repositories</li>
</ul>
</ul>
</ul>
<div>
<br /></div>
<ul style="text-align: left;">
<li>Define your Maven tool within the menu<i> Jenkins > Manage Jenkins > Global Tool </i>Configuration page</li>
<ul>
<li>Click on Maven Installations</li>
<ul>
<li>Specify name for your maven </li>
<li>Specify maven home directory e.g<i> /usr/local/maven-3.2.5</i></li>
<li>Check the Automatic install option</li>
<li>Choose Install from Apache e.g<i> maven-3.2.5</i></li>
</ul>
</ul>
</ul>
<div>
<br /></div>
<ul style="text-align: left;">
<li>Make sure that you Jenkins has access to install Maven within your maven home directory by executing the following command (on your slave ):</li>
<ul>
<li><span style="background-color: black;"><span style="color: white;">sudo chmod -R ugo+rw /usr/local/maven-3.2.5</span></span></li>
</ul>
</ul>
<div>
<br /></div>
<div>
<br /></div>
<ul style="text-align: left;">
<li>Now you can use maven in your Jenkins pipeline using a command such as :</li>
</ul>
<div>
<div>
<b>withMaven</b>(globalMavenSettingsConfig: 'maven-atlas-global-settings', jdk: 'JDK6', maven: 'M3_3.2.5', mavenLocalRepo: '/home/ubuntu/.m2/repository/') {</div>
<div>
</div>
<div>
sh 'mvn clean install ' </div>
<div>
</div>
<div>
}</div>
</div>
<div>
<br /></div>
<div>
Note that you can use the Pipeline Syntax helper to fill the options you want to use with Maven .</div>
<div>
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-2011620218842665932017-01-05T10:07:00.001+03:302017-01-05T10:11:31.966+03:30Publish Docker Image to Amazon ECR<div dir="ltr" style="text-align: left;" trbidi="on">
If you are using an Amazon AWS chances are that you already have ECR , Amazon EC2 Container Registry , within your account . Now this is practical if you want to have you own private Docker Registry for saving your docker images .<br />
<br />
Now in my case I wanted to be able to push an image to my private Registry within the context of a Jenkins build .<br />
<br />
So we will need to do the following :<br />
<br />
<ul style="text-align: left;">
<li>Configure AWS credentials on build machine</li>
<li>Configure Amazon ECR Docker Registry</li>
<li>Modify our Jenkins pipeline to perform a push </li>
</ul>
<br />
<div>
<br /></div>
<b><u>Configure AWS credentials on build machine</u></b><br />
<br />
1. install the awscli which allows you then to configure your aws account login info on your env , this is done using :<br />
<br />
<span style="background-color: black; color: white;">sudo apt install awscli</span><br />
<br />
2. next we do the aws configuration using the following command, ( <a href="http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html" target="_blank">see AWS CLI official guide</a> ):<br />
<br />
<span style="background-color: black;"><span style="color: white;">aws configure</span></span><br />
<div>
<br /></div>
<div>
Here you will need to know your AWS Access Key ID and AWS Secret Access Key .<br />
<br />
Note that the Secret Access Key ID is generated only once , so you need to keep it somewhere safe or regenerate a new one .<br />
<br />
To get the 2 keys you would need to login to your AWS console and go to :</div>
<div>
<br /></div>
<div>
<span style="background-color: black; color: cyan;">IAM > Users > Now select one of the users > Click on Security Credentials tab > Now from here you can create a New Access Key </span></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<b><u>Configure Amazon ECR Docker Registry</u></b></div>
<div>
<br /></div>
<div>
1. Login to your AWS console .</div>
<div>
2. Choose "EC2 Container Service"</div>
<div>
3. Click on Repositories > Create Repository</div>
<div>
4. Set a name for your repository </div>
<div>
5. Clicking on next will give you all the commands to login to ECR from aws cli , tag and push your image to your repo</div>
<div>
<br /></div>
<div>
For reference the official link to ECR is <a href="http://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_GetStarted.html" target="_blank">here</a> .</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><u>Modify our Jenkins pipeline to perform a push</u></b></div>
<div>
<br /></div>
<div>
<div>
Now that we have aws login configured on build machine and a private docker registry on Amazon we are ready to modify our Jenkins pipeline to perform the push .</div>
<div>
<br /></div>
<div>
Here I assume that you already do have Jenkins job existing and you know your way through the pipeline goovy codes .</div>
<div>
<br /></div>
<div>
So we will add the following :</div>
<div>
<br /></div>
<div>
{</div>
<div>
....</div>
<div>
}</div>
<div>
<div>
stage('Publish Docker Image to AWS ECR '){</div>
<div>
</div>
<div>
def loginAwsEcrInfo = <b>sh(returnStdout: true, script: 'aws ecr get-login --region us-east-1').trim()</b></div>
<div>
echo "Retreived AWS Login: ${loginAwsEcrInfo}"</div>
<div>
</div>
<div>
sh '${loginAwsEcrInfo}' </div>
<div>
sh 'docker <b>tag</b> tomcat6-atlas:latest XXXXXXXXXXXX.YYY.ZZZ.us-east-1.amazonaws.com/tomcat6-atlas:latest'</div>
<div>
sh 'docker <b>push</b> XXXXXXXXXXXX.YYY.ZZZ.us-east-1.amazonaws.com/tomcat6-atlas:latest'</div>
<div>
</div>
<div>
}</div>
</div>
<div>
<br /></div>
<div>
Note: Do replace the tag and push command with the actual values as indicated from your Amazon ECR repository page</div>
<div>
<br /></div>
<div>
Notice that I have a loginAwsEcrInfo variable defined in grovy , this is because I need to get the output of the command ' <b>aws ecr get-login --region us-east-1</b> ' from sh which actually generates the command to login through docker using the aws credentials . This is possible thanks to the <b>returnStdout</b> flag on sh .</div>
</div>
<div>
<br /></div>
<div>
That should be it , you should be able to publish your image within your Jenkins job execution .</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-2270256474253340312017-01-04T11:28:00.000+03:302017-01-04T14:40:20.103+03:30Linking Containers together using --link and Docker Compose<div dir="ltr" style="text-align: left;" trbidi="on">
Right now I am working on a project where :<br />
- there is a need for the tomcat instance to connect to an Oracle instance .<br />
- Both of these run in docker containers<br />
- I consider the Oracle instance to be a shared docker service , meaning it will be used by other services than the tomcat instance and that I do not want to tear it down as regularly as the tomcat docker instance<br />
<br />
I would first need to build an image of my webapp with tomcat6 using a command similar below :<br />
<br />
<span style="background-color: black; color: white;">docker build -t tomcat6-atlas .</span><br />
<br />
<br />
Then typically i use the following commands to run my docker image for tomcat:<br />
<br />
<span style="background-color: black; color: white;">docker run -it --rm <u>--link</u> <b>atlas_oracle12</b> --name <b>tomcat6-atlas-server</b> -p <b>8888:8080</b> <b>tomcat6-atlas</b></span><br />
<br />
This tells my docker that I want to :<br />
<br />
<ol style="text-align: left;">
<li> run an image of <b>tomcat6-atlas</b> as a container </li>
<li>the alias name of the container should be <b>tomcat6-atlas-server</b> using the --name flag</li>
<li>the port 8080 on the container should be mapped to 8888 on the host using -p flag</li>
<li>and that i should link my<b> atlas_oracle12</b> container which is already started ( check this <a href="http://javedmandary.blogspot.com/2016/12/oracle-12c-setup-using-docker-on-ubuntu.html" target="_blank">blog entry</a> ) to this tomcat6-atlas-server container that am firing using the <u>--link</u> flag . </li>
</ol>
<div>
The --link flag is important because using this , I can specify for exampled the JDBC connection from my app in the <b>tomcat6-atlas-server </b>container to point to the atlas_oracle12 container using the alias name directly instead of having to use some ip addresses ( which may change if I restart the oracle container ) .</div>
<div>
<br /></div>
<div>
You could actualy ping the atlas_oracle12 container from the tomcat6-atlas container just by doing ping atlas_oracle12 , you dont need to therefore know the ip address of atlas_oracle12 as long as you name what is the alias name of the container .</div>
<div>
<br /></div>
<div>
<b><u>Docker Compose </u></b></div>
<div>
<br /></div>
<div>
Now typically the above is great if you have a small project but assume that the tomcat6-atlas container had numerous dependencies with other containers then it the command quickly becomes quite volumetric and possibly error prone.</div>
<div>
<br /></div>
<div>
Here comes <a href="https://docs.docker.com/compose/" target="_blank">Docker Compose</a> which simplifies the build and the run of the container using one yml /yaml file as shown below:</div>
<div>
<br /></div>
<br />
<div>
<div>
version: '2'</div>
<div>
services:</div>
<div>
atlas_tomcat6:</div>
<div>
build: .</div>
<div>
build:</div>
<div>
context: .</div>
<div>
dockerfile: Dockerfile</div>
<div>
image: tomcat6-atlas:latest</div>
<div>
</div>
<div>
network_mode: bridge</div>
<div>
<br /></div>
<div>
external_links:</div>
<div>
- atlas_oracle12</div>
<div>
</div>
<div>
ports:</div>
<div>
- 8888:8080</div>
<div>
privileged: true</div>
<div>
</div>
</div>
<div>
This is typically written in a <b>docker-compose.yml</b> file and you need to also <a href="https://docs.docker.com/compose/install/" target="_blank">install Docker Compose</a>. </div>
<div>
<br /></div>
<div>
Important things is that :</div>
<div>
<ol style="text-align: left;">
<li>It specifies the name of the project as atlas_tomcat6</li>
<li>It assumes that in the same location as the docker-compose.yml file there is a Dockerfile to perform the build</li>
<li>It knows thats the name and tag of the image is 'tomcat6-atlas' and 'latest' respectively</li>
<li>With the <b>network_mode:bridge</b> value it understands that instead of creating a seperate network for the docker compose triggered instance of the container that it needs to use the default network of the host bridge , that is it will be able to connect to<b> atlas_oracle12</b> ( container which was not started by docker compose )</li>
<li>Containers on which atlas_tomcat6 has a dependency on but triggered seperately are defined with external-links tag e.g atlas_oracle12</li>
<li>ports tag specifies the port mappings</li>
</ol>
<div>
I can build an image for tomcat6-atlas using the command :<br />
<br />
<span style="background-color: black; color: white;">docker-compose build</span><br />
<br />
<br /></div>
</div>
<div>
Now all you need to do is to fire up docker-compose using :</div>
<div>
<br /></div>
<div>
<span style="background-color: black; color: white;">docker-compose up</span></div>
<div>
<br /></div>
<div>
Note that if the previous build command was not executed as part of the up command the image would first be built and then started.<br />
<br />
If you want to run this in the bakground then you an use the -d flag :</div>
<div>
<br /></div>
<div>
<span style="background-color: black; color: white;">docker-compose up -d</span></div>
<div>
<br /></div>
<div>
To shut down your containers just use :</div>
<div>
<br /></div>
<div>
<span style="background-color: black; color: white;">docker-compose down </span></div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-47338674783701476812017-01-04T10:53:00.001+03:302017-01-16T17:05:00.417+03:30Portainer for visualizing your docker infra<div dir="ltr" style="text-align: left;" trbidi="on">
So after having played around with <a href="http://javedmandary.blogspot.com/2016/12/install-shipyard-to-monitor-docker.html" target="_blank">shipyard </a> I decided to give <a href="http://portainer.io/" target="_blank">Portainer </a>a try . The reason why I wanted to look at Portainer as it gives you much more information around your Docker infra than shipyard does.<br />
<br />
Below is a screenshot showing the features within shipyard:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-7mOO3ABhcME/WGygOMmFCVI/AAAAAAAAask/oD6qFHgU9Rk_3WRfT3nI48MiwW6_HVNuQCLcB/s1600/shipyard.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="273" src="https://2.bp.blogspot.com/-7mOO3ABhcME/WGygOMmFCVI/AAAAAAAAask/oD6qFHgU9Rk_3WRfT3nI48MiwW6_HVNuQCLcB/s640/shipyard.PNG" width="640" /></a></div>
<br />
You can see that it has information around containers , images , nodes and registries and that pretty much stops there.<br />
<br />
In comparison Portainer provides much more level of details :<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-dr4gYOBzf_k/WGygldHe3kI/AAAAAAAAaso/12eH19SKSQkV2hbM2nFQHjagE8bU46reQCLcB/s1600/portainer.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="360" src="https://2.bp.blogspot.com/-dr4gYOBzf_k/WGygldHe3kI/AAAAAAAAaso/12eH19SKSQkV2hbM2nFQHjagE8bU46reQCLcB/s640/portainer.PNG" width="640" /></a></div>
<br />
The thing that interested the most was the Networks section as I was trying to figure out how to connect a docker-compose tiggered container with a shared container which was not launched through docker-compose.<br />
<br />
<b><u>Installation Portainer</u></b> :<br />
<br />
- You need as pre-requiste to have docker and docker swarm installed<br />
- Official installation instructions are <a href="http://portainer.io/install.html" target="_blank">here </a><br />
- Then just execute the following command to install the Portainer container ,which will be exposed on port 9000 :<br />
<br />
<span style="background-color: black; color: white; font-family: "courier new" , monospace; font-size: 14.04px; white-space: pre;">docker run -d -p 9000:9000 portainer/portainer</span><br />
<span style="background-color: rgba(255 , 255 , 255 , 0.0235294); font-family: "courier new" , monospace; font-size: 14.04px; white-space: pre;"><br /></span>
<span style="background-color: rgba(255 , 255 , 255 , 0.0235294); font-family: "courier new" , monospace; font-size: 14.04px; white-space: pre;">Note that am assuming that your running on ubuntu /linux </span><br />
<br />
To run portainer on a local instance of the docker engine use the following command :<br />
<br />
<span style="background-color: black;"><span style="color: white;">docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer</span></span><br />
<br />
<b><u>Endpoints</u></b><br />
<br />
You an have multiple endpoints configured such as you are monitoring diferent remote instances :<br />
- make sure that inbound ports are opened on your remote endpoints (e.g 2375 )<br />
- if you run Portainer locally to your docker containers , there is a recommended setting to be changed or just provide the public ip addr. of the docker host </div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-58134601219592066052016-12-29T15:00:00.002+03:302017-01-04T10:54:08.336+03:30Install Shipyard to monitor Docker Containers<div dir="ltr" style="text-align: left;" trbidi="on">
So far I have a number of containers on my ubuntu box , looked at the easiest way to manage them all and gave <a href="https://shipyard-project.com/" target="_blank">shipyard </a>a try .<br />
<br />
There are 2 ways to install shipyard both involve (without any surprise) to make use of docker containers.<br />
<br />
I have tried the manual install as it gives me more flexibility . The link to the installation is found <a href="https://shipyard-project.com/docs/deploy/manual/" target="_blank">here</a> . This comes as a number of docker images to run .<br />
<br />
The key thing to bear in mind is that wherever you see <IP-OF-HOST> you need to add the actual <b>public ip address</b> of the docker host .<br />
<br />
Below are some examples of where the swarm manager and agent ask for the <IP_OF_HOST><br />
<br />
<span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">docker run \
-ti \
</span><span class="hljs-operator" style="box-sizing: border-box; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">-d</span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"> \
--restart=always \
--name shipyard-swarm-manager \
swarm:latest \ </span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"> manage --host tcp://</span><span style="background-color: black;"><span class="hljs-number" style="box-sizing: border-box; color: #2aa198; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">0.0</span><span style="color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">.</span><span class="hljs-number" style="box-sizing: border-box; color: #2aa198; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">0.0</span><span style="color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">:</span><span class="hljs-number" style="box-sizing: border-box; color: #2aa198; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">3375</span></span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"> etcd://<IP-OF-HOST>:</span><span class="hljs-number" style="box-sizing: border-box; color: #2aa198; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">4001</span><br />
<br />
<span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">docker run \
-ti \
</span><span class="hljs-operator" style="box-sizing: border-box; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">-d</span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"> \
--restart=always \
--name shipyard-swarm-agent \
swarm:latest \ </span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"> join --addr <ip-of-host>:</span><span class="hljs-number" style="box-sizing: border-box; color: #2aa198; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">2375</span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"> etcd://<ip-of-host>:</span><span class="hljs-number" style="box-sizing: border-box; color: #2aa198; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">4001</span><br />
<br />
Do not put localhost as this IP address else you will not be able to view containers on the docker host .<br />
<br />
Also you can configure the port on which your shipyard WEB GUI is accessible by changing the port number below highlighted in yelow i.e 7777<br />
<br />
<span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">docker run \
-ti \
</span><span class="hljs-operator" style="box-sizing: border-box; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">-d</span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"> \
--restart=always \
--name shipyard-controller \
--link shipyard-rethinkdb:rethinkdb \
--link shipyard-swarm-manager:swarm \
-p </span><span style="background-color: orange; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"><span style="color: #444444;">7777</span></span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">:</span><span class="hljs-number" style="box-sizing: border-box; color: #2aa198; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">8080</span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"> \
shipyard/shipyard:latest \
server \
</span><span class="hljs-operator" style="box-sizing: border-box; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">-d</span><span style="background-color: #002b36; color: #839496; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;"> tcp://swarm:</span><span class="hljs-number" style="box-sizing: border-box; color: #2aa198; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 16px; white-space: pre-wrap;">3375</span></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-24519016850658812382016-12-27T17:13:00.000+03:302017-01-04T10:54:32.905+03:30Oracle 12c setup using Docker on Ubuntu<div dir="ltr" style="text-align: left;" trbidi="on">
Recently had to install Oracle 12c on an ubuntu 16.04 server , the quickest way I found to do that was through docker .<br />
<br />
<b><u>Pre-requisites</u></b><br />
<br />
First things first we need to setup docker and this is done by following the docker docs :<br />
<a href="https://docs.docker.com/engine/installation/linux/ubuntulinux/">https://docs.docker.com/engine/installation/linux/ubuntulinux/</a><br />
<br />
<br />
<b><u>Installing Oracle Image </u></b><br />
<br />
Now that you have a working docker install you need to :<br />
<br />
<ol style="text-align: left;">
<li>Download the image for Oracle 12c </li>
<li>Open port 8080 and 1541 such as you get access to the web Application express interface and are able to connect to the Oracle instance via SQLPlus respectively</li>
<li>Map a source directory on your docker host with a directory within the Docker Oracle container should you want to import dumps for example</li>
</ol>
<div>
All the above can be achieved within the command below :</div>
<div>
docker run -d -P --name <b>atlas_oracle12</b> -p 8080:8080 -p 1521:1521 -v <b>/home/ubuntu/atlaslocaldump<span style="color: red;">:</span><span style="color: blue;">/u01/app/oracle/admin/xe/dpdump/</span></b> <b>sath89/oracle-12c </b> </div>
<div>
<br /></div>
<div>
Things to note are that :</div>
<div>
<ol style="text-align: left;">
<li><b>atlas_oracle12 </b>- this is the name I have given to my container , it can be any valid name e.g foo </li>
<li><b>/home/ubuntu/atlaslocaldump - </b>this directory on my docker host which i want to make visible within the oracle docker container ( so basically the source )</li>
<li><span style="color: blue; font-weight: bold;">/u01/app/oracle/admin/xe/dpdump/ </span>- this is the directory on the docker container from which i will be able to access the files within <b>/home/ubuntu/atlaslocaldump </b></li>
<li><b style="font-weight: bold;">sath89/oracle-12c -</b> this is the name of the image for the Oracle-12c install , you can get more information around this<a href="https://hub.docker.com/r/sath89/oracle-12c/" target="_blank"> here on docker hub</a> .</li>
<li>Also it takes around 10-15 mins depending on your machine to initialise the Oracle instance so you might not just be able connect straight to SQLPlus .. give it some time to initialise</li>
</ol>
<div>
<b><u>SQLPlus</u></b></div>
<div>
<b><u><br /></u></b></div>
<div>
So once the DB is up and running you might want to access the Oracle instance via SQLPlus , to do that you can either install SQLPlus on your docker host and connect or you go within your Oracle container and access the bash , I have done the later as installation of SQLPlus client on Ubuntu was a completely nightmare .</div>
<div>
<br /></div>
<div>
So connect to the oracle container using the following command :</div>
<div>
docker exec -it <b>atlas_oracle12</b> bash </div>
<div>
<br /></div>
<div>
Note that <b>atlas_oracle12</b> is the name of the container that you defined in the docker run command above if this is not the name of the container then change it to reflect your own container name.</div>
</div>
<div>
<br /></div>
<div>
Now that we are within the container SQLPlus can be called using :</div>
<div>
$ORACLE_HOME/bin/sqlplus system/oracle@//localhost:1521/xe.oracle.docker</div>
<div>
<br /></div>
<div>
<b><u>Importing a Dump</u></b></div>
<div>
<br /></div>
<div>
You can also import a dump using the following command :</div>
<br />
<div>
$ORACLE_HOME/bin/impdp<b> USERNAME/PASSWORD</b>@//localhost:1521/xe.oracle.docker dumpfile=<b>myDumpFile.dmp </b> logfile=<b>myLogFile</b>.log table_exists_action=replace schemas=<b>mySchema</b> </div>
<div>
<div>
</div>
</div>
<div>
Do change the values above to correspond to your specific settings<br />
<br />
<b><u>Removing the Oracle Container</u></b></div>
<br />
For some reasons you might want to remove the oracle container in our case it is named <b>atlas_oracle12</b> ( change this below to the name you gave your container instance )<br />
<br />
To do that you need to stop the contained using command :<br />
<br />
docker stop <b>atlas_oracle12</b><br />
<br />
Then remove the container directly using :<br />
<br />
docker rm <b>atlas_oracle12</b><br />
<br />
You can check that the container is removed by doing a :<br />
<br />
docker ps<br />
<div>
<br /></div>
<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-68540187899192816562016-11-13T22:03:00.000+03:302017-01-04T11:35:29.296+03:30Truffle and Ethereum - call vs transactions<div dir="ltr" style="text-align: left;" trbidi="on">
After banging my head for 4 hours trying to figure out why my sample Dapp doesn't actually save (persist) data to my contract I found out that i was actually making a <b>call</b> instead of executing a <b>transaction</b> .<br />
<br />
What's the difference between a call and a transaction you say ?<br />
<br />
- Well a <b><u>call</u></b> is <b>free</b> ( does not cost any gas ) , it is <b>readonly</b> and will <b>return</b> a value<br />
- whilst a <b><u>transaction</u></b> <b>costs</b> gas , it usually <b>persists</b> something and <b>does not return</b> anything<br />
<br />
Please read the "Making a transaction" and "Making a call" section in the offical lib:<br />
<a href="https://truffle.readthedocs.io/en/latest/getting_started/contracts/" target="_blank">https://truffle.readthedocs.io/en/latest/getting_started/contracts/ </a><br />
<br />
Say for example in your app.js of your truffle app you have a fictionious setComment and getComment function ( to persist and retrieve comments respectively ) this is how they could look like :<br />
<br />
===Extract app.js====================<br />
<br />
function setComment(){<br />
<br />
<span style="color: blue;"> // Assuming your contract is called MyContract</span><br />
var meta = MyContract.deployed();<br />
<br />
<span style="color: blue;">//Taking a comment field from page to save </span><br />
var comment = document.getElementById("comment").value;<br />
<br />
<span style="color: blue;">// Note that the setComment should exist in your MyContract</span><br />
meta.<b><u>setComment</u></b>(comment,{from: account}).then(function() {<br />
<br />
setStatus( "Tried to save Comment:" +comment);<br />
}).catch(function(e) {<br />
console.log(e);<br />
setStatus("Error saving comment .. see log."+ e);<br />
});<br />
}<br />
<br />
<br />
<br />
function getComment(){<br />
var meta = MyContract.deployed();<br />
<br />
<span style="color: lime;"> </span><span style="color: blue;">// notice that in the readonly getComment method that after method name a <b>.call </b>is used</span><br />
<span style="color: blue;">// this is what differentiates a call from a transaction</span><br />
meta.<b><u>getComment</u></b>.<b>call</b>({from: account}).then(function(value) {<br />
<br />
setStatus( "Comment:" +value);<br />
}).catch(function(e) {<br />
console.log(e);<br />
setStatus("Error retrieving comment .. see log."+ e);<br />
});<br />
}<br />
<br />
=============================<br />
<br />
<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com2tag:blogger.com,1999:blog-8096130.post-58823033576390938622016-11-02T17:38:00.000+03:302017-01-04T11:35:51.734+03:30Ethereum Blockchain Simple Contract Example<div dir="ltr" style="text-align: left;" trbidi="on">
This article is about how to get started with creating and deploying a really basic smart contract on the <a href="https://www.ethereum.org/" target="_blank">Ethereum </a>network using the <a href="https://truffle.readthedocs.io/en/latest/" target="_blank">Truffle </a>development environment and some solidity scripting .<br />
<br />
<a href="https://solidity.readthedocs.io/en/develop/" target="_blank">Solidity </a>is the language in which the contracts are writing its pretty simple enough but lets make some small steps against which then we can build more complex scenarios.<br />
<br />
<b>Assumptions :</b><br />
<br />
<br />
<ul style="text-align: left;">
<li>We are deploying to an Ubuntu env. , although the commands could be adapted to whichever environment your using </li>
<li>We are starting with a fresh Ubuntu with nothing on it</li>
<li>Ubuntu version used is 16.04 LTS</li>
<li>We are going to use a development framework such as Truffle to help us in quickly testing our smart contract</li>
<li>We are not going to connect to either the live or test 'morden' Ethereum network instead we are going to run a test environment locally, this is done via the <a href="https://github.com/ethereumjs/testrpc" target="_blank">ethereumjs-testrpc</a></li>
<li>Ubuntu server in my case is a t2.medium server 2 cpu with 4GB RAM at least </li>
</ul>
<div>
<br /></div>
<div>
<b><u>Prerequisites</u></b></div>
<div>
<br /></div>
<div>
<b>- NodeJS 6</b></div>
<div>
<b><br /></b></div>
<div>
Execute the following commands :</div>
<div>
<br /></div>
<br />
<div>
sudo apt-get install python-software-properties</div>
<div>
<div>
$ curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -</div>
</div>
<div>
<div>
sudo apt-get install -y nodejs</div>
<div>
sudo apt-get -y install build-essential</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<div>
<b>-Install GIT:</b></div>
<div>
sudo apt-get -y install git</div>
</div>
<div>
<br /></div>
<div>
<div>
<b>- Install truffle:</b></div>
<div>
sudo npm install -g truffle</div>
</div>
<div>
<br /></div>
<div>
<b>- Install euthereumjs-testrpc:</b></div>
<div>
sudo npm install -g ethereumjs-testrpc</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><u>Starting Test Ethereum Node</u></b></div>
<div>
<br /></div>
<div>
Assuming all the installs above were executed without any issues , the first thing to do is fire up your test rpc server using the following command :</div>
<div>
<span style="color: orange;"><br /></span></div>
<div>
<span style="color: blue;">testrpc</span></div>
<div>
<br /></div>
<div>
This should output something like this:</div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-_bbJz4Q45JU/WBnqRiB4MOI/AAAAAAAAZNk/FZhRVRG_6ggrMpCEhJk6ourXjShP9peOQCLcB/s1600/testrpc_startup.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="524" src="https://4.bp.blogspot.com/-_bbJz4Q45JU/WBnqRiB4MOI/AAAAAAAAZNk/FZhRVRG_6ggrMpCEhJk6ourXjShP9peOQCLcB/s640/testrpc_startup.PNG" width="640" /></a></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><u>Truffle Workspace</u></b></div>
<div>
<br /></div>
<div>
Now lets follow the example from the truffle website on <a href="https://truffle.readthedocs.io/en/latest/getting_started/project/" target="_blank">getting started</a> :</div>
<div>
<br /></div>
<div>
1. Create a directory myproject : mkdir myproject </div>
<div>
2. cd to the directory : cd myproject</div>
<div>
3. Execute the truffle initialisation command : truffle init</div>
<div>
<br /></div>
<div>
This will generate a number of truffle files to ease our development to check the official documentation to stay up to date with more extensive information around what each folder and each file does .</div>
<div>
<br /></div>
<div>
However we are concerned at the moment only with the contracts and this would be in the contracts folder where you will see the following files:</div>
<div>
<span style="color: blue;">ConvertLib.sol MetaCoin.sol Migrations.sol</span></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
The MetaCoin.sol is the main contract , it imports the ConvertLib.sol library and the Migrations.sol tells truffle how the contracts needs to be deployed. Feel free to open them to have a look.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><u>Adding our own contrats</u></b> </div>
<div>
<br /></div>
<div>
Now in that same directory we are going to add 2 new sol files for a very simple dummy hello world type application ( contract name and file name needs to be exactly the same ) :</div>
<div>
<br /></div>
<div>
<span style="color: #0b5394;">-------Contents of Mortal.sol----------</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<div>
<span style="color: #0b5394;">pragma solidity ^0.4.4;</span></div>
<div>
<span style="color: #0b5394;">contract Mortal{</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;"> address public owner;</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;"> function mortal(){</span></div>
<div>
<span style="color: #0b5394;"> owner = msg.sender;</span></div>
<div>
<span style="color: #0b5394;"> }</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;"> function kill(){</span></div>
<div>
<span style="color: #0b5394;"> suicide(owner);</span></div>
<div>
<span style="color: #0b5394;"> }</span></div>
<div>
<span style="color: #0b5394;">}</span></div>
</div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;">-----------------</span></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Then another sol file</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<span style="color: #0b5394;">-----User.sol-----------------</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<div>
<span style="color: #0b5394;">pragma solidity ^0.4.4;</span></div>
<div>
<b><span style="color: #0b5394;">import "Mortal.sol";</span></b></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;">contract User is Mortal{</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;"> string public userName;</span></div>
<div>
<span style="color: #0b5394;"> string public ourName;</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;"> function User(){</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;"> userName= "Javed";</span></div>
<div>
<span style="color: #0b5394;"> }</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;"> function hello() constant returns(string){</span></div>
<div>
<span style="color: #0b5394;"> return "Hello There !";</span></div>
<div>
<span style="color: #0b5394;"> }</span></div>
<div>
<span style="color: #0b5394;"><br /></span></div>
<div>
<span style="color: #0b5394;"> function getUserName() constant returns(string){</span></div>
<div>
<span style="color: #0b5394;"> return userName;</span></div>
<div>
<span style="color: #0b5394;"> }</span></div>
<div>
<span style="color: #0b5394;">}</span></div>
</div>
<div>
<span style="color: #0b5394;">-----------------------------------</span></div>
<div>
<br /></div>
<div>
If you want to know what these files are doing along with syntax i suggest that you follow this great video from one awesome guy that allowed me to get started quite quickly here are his videos you should watch ( subscribe to his channel ) :</div>
<div>
<br /></div>
<div>
1. <a href="https://www.youtube.com/watch?v=9_coM_g7Dbg" target="_blank">Smart Contract 1</a></div>
<div>
2. <a href="https://www.youtube.com/watch?v=HAvDbKttijY" target="_blank">Smart Contract 2</a></div>
<div>
3. <a href="https://www.youtube.com/watch?v=FGnv8Vfu9bY" target="_blank">Smart Contract 3</a></div>
<div>
<br /></div>
<div>
In our case we modified his example such as we get something we can very quickly run and see what happens :) !!</div>
<div>
<br /></div>
<div>
Also in his example he uses the Ethereum wallet but this was a very process for me as the wallet keep synching for a very long time and not even sure if its working fine as well , so better start with a test env . with <b>testrpc </b>.</div>
<div>
<br /></div>
<div>
<u><b>Truffle Compile</b></u></div>
<div>
<br /></div>
<div>
Now go on the console and run the <a href="https://truffle.readthedocs.io/en/latest/getting_started/compile/" target="_blank">compile command</a> : </div>
<div>
<br /></div>
<div>
<span style="color: #0b5394;">truffle compile</span></div>
<div>
<br /></div>
<div>
This will compile your contracts in bytecode .</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><u>Truffle Migrate</u></b></div>
<div>
<br /></div>
<div>
Now lets move the compiled contracts to test env. ethereum node that is testrcp :</div>
<div>
<br /></div>
<div>
truffle migrate</div>
<div>
<br /></div>
<div>
You should see something like in the left window :</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-MLWL48c6Q4U/WBnw2eFjJrI/AAAAAAAAZN0/O41n685bH3k9PsPRc7mPy3X7o4L4EMPFwCLcB/s1600/compile.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="202" src="https://2.bp.blogspot.com/-MLWL48c6Q4U/WBnw2eFjJrI/AAAAAAAAZN0/O41n685bH3k9PsPRc7mPy3X7o4L4EMPFwCLcB/s640/compile.PNG" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><u><br /></u></b></div>
<div>
<b><u>Truffle Build</u></b></div>
<div>
<br /></div>
<div>
Now you can build a dapp ( distributed application ) using truffle , i wont go in the details of this as you will find more information from the <a href="https://truffle.readthedocs.io/en/latest/getting_started/build/" target="_blank">main documentation page</a> and i am only going to get you started quickly.</div>
<div>
<br /></div>
<div>
Command is well :</div>
<div>
<br /></div>
<div>
<span style="color: #0b5394;">truffle build</span></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><u>Truffle Console</u></b></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Truffle comes with a console that allows you to tap directly into your deployed contract so what you want to do is fire up the console using the command :</div>
<div>
<br /></div>
<div>
truffle console </div>
<div>
<br /></div>
<div>
Try to execute the following commands :</div>
<div>
<br /></div>
<div>
User.deployed();</div>
<div>
<br /></div>
<div>
User.deployed().<b>userName</b>.call();</div>
<div>
<br /></div>
<div>
User.deployed().<b>hello</b>.call();</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
You should be able to retrieve values from public property userName and execute function hello.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-chKcKn9J1IY/WBnyIftMRPI/AAAAAAAAZN8/lFzxZLb3x4kzKBxs9wK-LmP0Y98Dyq-kACLcB/s1600/console.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="398" src="https://1.bp.blogspot.com/-chKcKn9J1IY/WBnyIftMRPI/AAAAAAAAZN8/lFzxZLb3x4kzKBxs9wK-LmP0Y98Dyq-kACLcB/s640/console.PNG" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><u>Conclusion</u></b></div>
<div>
<b><u><br /></u></b></div>
<div>
We managed to :</div>
<div>
1. Setup an ubuntu server with all the tools we need to start developping dapps ( nodejs , truffle , ethereumjs-testrpc )</div>
<div>
2. We managed to learn about the development framework truffle how to use it</div>
<div>
3. We created , compiled and deployed a simple dummy contract to the testrpc node</div>
<div>
4. We managed to even call the contract </div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com1tag:blogger.com,1999:blog-8096130.post-88901495199300162162016-10-24T18:24:00.001+03:302017-01-04T11:36:09.661+03:30First Steps with Blockchain<div dir="ltr" style="text-align: left;" trbidi="on">
Blockchain can be very confusing to start with so I will be documenting my findings on my blog such as to keep a trace of how to set it up and get started creating decentralized apps also known as <b>dapp</b> .<br />
<div>
<br /></div>
<div>
I have purchased a book which is really interesting and that explains where we are currently as far as cytocurrencies are (e.g BitCoin , ether , litecoin ) but also the history behind blockchain. The book is called <a href="https://www.amazon.com/Decentralized-Applications-Harnessing-Blockchain-Technology/dp/1491924543/ref=sr_1_1?ie=UTF8&qid=1477320115&sr=8-1&keywords=decentralized+applications+blockchain" target="_blank">Decentralized Applications</a> written by Siraj Raval . I really appreciated the way that the author explained the concepts in a simple but yet extensive manner , only problem I got was that when I came to chapter 3 where the author basically references open source github project to create a dapp that basically mimics twitter , the github urls are all broken hence the application example is really hard to follow.</div>
<div>
<br /></div>
<div>
Now currently trying to make the hello world application for Ethereum work , steps have been explained on this blog post although running a contract seems to be churning out all kind of errors:</div>
<div>
<a href="http://vanderwijk.info/blog/getting-started-ethereum/">http://vanderwijk.info/blog/getting-started-ethereum/</a> </div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-33387583119512102462016-09-19T19:51:00.004+04:302016-09-19T20:20:01.875+04:30Installing mvnw on Ubuntu<div dir="ltr" style="text-align: left;" trbidi="on">
Typically when working with jhipster applications there is are mvnw commands that needs to be executed.<br />
<br />
The mvnw tool which is wrapper on top of maven needs to be installed by following the example set at <a href="https://github.com/vdemeester/mvnw">https://github.com/vdemeester/mvnw</a> .<br />
<br />
First need to checkout the tool :<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">git clone git://github.com/vdemeester/mvnw.git <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">~</span>/.mvnw</pre>
<br />
Then you need to add the following environment variable within .bashrc:<br />
<br />
<br style="background-color: white; color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13px;" /><span style="background-color: black; color: white; font-family: monospace; font-size: 12.1875px;">nano ~/.bashrc</span><br />
<br />
Command:<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-s" style="box-sizing: border-box; color: #183691;">export PATH="$HOME/.mvnw/bin:$PATH"</span></pre>
<br />
then enable it :<br />
<span style="background-color: black; color: white; font-family: monospace; font-size: 12.1875px;">. ~/.bashrc</span><br />
<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-37306814216440273882016-09-19T19:51:00.003+04:302016-09-19T19:54:48.019+04:30Install Maven 3.3.9 on ubuntu<div dir="ltr" style="text-align: left;" trbidi="on">
The following commands needs to be adapted based on the version of Maven you want to install , the latest version can be found <a href="https://maven.apache.org/download.cgi" target="_blank">on Maven download page</a> .<br />
<br />
This article assumes that you are installing on Ubuntu the version 3.3.9 of maven ( currently latest version as am writing entry ):<br />
<br />
<span style="background-color: black;"><span style="color: white;">wget http://apache.mirrors.lucidnetworks.net/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz</span></span><br />
<br />
<span style="background-color: black; color: white;">sudo mkdir -p /usr/local/apache-maven</span><br />
<br />
<span style="background-color: black; color: white;">sudo mv apache-maven-3.3.9-bin.tar.gz /usr/local/apache-maven</span><br />
<br />
<span style="background-color: black;"><span style="color: white;">cd /usr/local/apache-maven</span></span><br />
<br />
<span style="background-color: black; color: white;">sudo tar -xzvf apache-maven-3.3.9-bin.tar.gz</span><br />
<br />
Once this is done add to you environment variables by editing .bashrc:<br />
<br />
<span style="background-color: black; color: white; font-family: monospace; font-size: 12.1875px;">nano ~/.bashrc</span><br />
<span style="background-color: black; color: white; font-family: monospace; font-size: 12.1875px;"><br /></span>
<span style="background-color: black;"><span style="color: white;">export M2_HOME=/usr/local/apache-maven/apache-maven-3.3.9</span></span><br />
<span style="background-color: black;"><span style="color: white;">export M2=$M2_HOME/bin</span></span><br />
<span style="background-color: black;"><span style="color: white;">export MAVEN_OPTS="-Xms256m -Xmx512m"</span></span><br />
<span style="background-color: black;"><span style="color: white;">export PATH=$M2:$PATH</span></span><br />
<br />
then apply it by executing :<br />
<span style="background-color: black; color: white; font-family: monospace; font-size: 12.1875px;">. ~/.bashrc</span><br />
<br />
<br />
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span>
<span style="background-color: black; color: white;"><br /></span></div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0tag:blogger.com,1999:blog-8096130.post-10569144319301579512016-08-21T21:20:00.003+04:302016-08-21T21:24:40.440+04:30Skytravel Agency<div dir="ltr" style="text-align: left;" trbidi="on">
One of my friends recently opened up a travel agency in Mauritius called <a href="http://skytime.mu/" target="_blank">SkyTime</a> , they have excellent rates and wonderful ideas for your next trip. So if you want to book your next vacations abroad do not hesitate to at least get a quote from these wonderful people who are passionate about their jobs.<br />
<div>
<br /></div>
<div>
Right now they are running some interesting promotions on trips to Kuala Lumpur , Singapore and Bangkok , don't miss these awesome deals !<br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-14Dyb3_TVYw/V7nbSEHugjI/AAAAAAAAYkI/eWSzlrQ7AikuxGR3FYTqYapfA3bo2HI6gCLcB/s1600/skytravel.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://3.bp.blogspot.com/-14Dyb3_TVYw/V7nbSEHugjI/AAAAAAAAYkI/eWSzlrQ7AikuxGR3FYTqYapfA3bo2HI6gCLcB/s640/skytravel.PNG" width="451" /></a></div>
<div>
<div style="text-align: center;">
<br />
<span style="background-color: white; color: #666666; font-family: "helvetica" , "arial" , sans-serif; font-size: 12px; line-height: 16.08px;">Airfare Promo to Singapore/Kuala Lumpur ***Rs 17,800***</span><br />
<span style="background-color: white; color: #666666; font-family: "helvetica" , "arial" , sans-serif; font-size: 12px; line-height: 16.08px;">4 Nights in Kuala Lumpur (3 star hotel) ***Rs 24,000***</span><br />
<span style="background-color: white; color: #666666; font-family: "helvetica" , "arial" , sans-serif; font-size: 12px; line-height: 16.08px;">4 Nights</span><span class="text_exposed_show" style="background-color: white; color: #666666; display: inline; font-family: "helvetica" , "arial" , sans-serif; font-size: 12px; line-height: 16.08px;"> in Kuala Lumpur (4 star hotel) ***Rs 26,000***</span><br />
<span style="background-color: white; color: #666666; font-family: "helvetica" , "arial" , sans-serif; font-size: 12px; line-height: 16.08px;">4 Nights in KL, 2 Nights in Singapore ***Rs 31,500***</span><br />
<span style="background-color: white; color: #666666; font-family: "helvetica" , "arial" , sans-serif; font-size: 12px; line-height: 16.08px;">3N in KL, 2N in Singapore, 3N in Bangkok ***Rs 45,000***</span><br />
<span style="background-color: white; color: #666666; font-family: "helvetica" , "arial" , sans-serif; font-size: 12px; line-height: 16.08px;">3N in KL, 2N in Singapore, 3N in Bali ***Rs 43,500***</span></div>
</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/08024494303633934159noreply@blogger.com0