IOS implementation of class Prisma software

Preface


Prisma on the 2016 line after the fire, the APP is the use of artificial neural networks and artificial intelligence technology, for ordinary photos to add artistic effect of the photo editing software.

The same year, Google also issued a “A LEARNED REPRESENTATION FOR ARTISTIC STYLE”, to realize the operation of an integration of a variety of artistic style for photos, and optimize the use of memory and computing speed, can fast operation on mobile devices.

Recently in the study of the integration of iOS Tensorflow, found that Google discloses the source code and training data, that is to say we can write a prior to the operation, application and integration of the training parameters can quickly achieve class Prisma.

Here’s how to run a iPhone on their own Prisma”.

IOS implementation of class Prisma software
money and grunt

Dead work


  1. Install Tensorflow, the official website of the detailed tutorial here is not much to say.
  2. Build iOS+Tensorflow project, which can be achieved on the basis of the steps of Git, you can refer to the official Demo program configuration. There are a lot of pits in this process, multiple attempts should be configured successfully
  3. Download model, the use of the model is image_stylization, Google has been open source on the GitHub.
  4. Download the training parameters, Google provides 2:
    Monet Varied
    Monet training of 10 kinds of art pictures, Varied training of the 32 kinds of
    .
    of course, you can also train their own art pictures, but have to download the VGG training parameters and ImageNet data, and then their own training, compare the time spent.

Construction calculation chart


Although Google provides a model of the source, but not the source output operational pattern has been easy to transfer to the use of mobile devices, the Android Demo is provided to generate the Pb, how to think to write their own calculation diagram can be directly copied to your trouble in engineering iOS.

I have created a python project, and then the Google source code model.py related documents are added to the project.
my building code is as follows:

Import numpy as NP import tensorflow as TF import ast import OS from tensorflow.python import pywrap_tensorflow from Matplotlib import pyplot from matplotlib.pyplot import imshow import image_utils import model import OPS import argparse import sys num_styles = 32 imgWidth = 512 imgHeight = 512 channel = 3 checkpoint = /Users/Jiao/Desktop/TFProject/style-image/checkpoint/multistyle-pastiche-generator-varied.ckpt inputImage = tf.placeholder (tf.float32, shape=[None, imgWidth, imgHeight, channel] name=, "input") styles = tf.placeholder (tf.float32, shape=[num_styles], name= "style" with ("tf.name_scope")): Transform (inputImage, normalizer_fn=ops.weighted_instance_norm = model.transform, Normalizer_params={#:'weights'tf.constant (mixture),'weights': styles,'num_categories': num_styles,'center': True,'scale': True}) model_saver = tf.train.Saver (tf.global_variables) (with) (tf.Session) as sess: tf.train.write_graph (sess.graph_def, "/Users/Jiao/Desktop/TFProject/style-image/protobuf", "input.pb") #checkpoint = os.path.expanduser (checkpoint) #if tf.gfile.IsDirectory (checkpoint): # checkpoint = tf.train.latest_checkpoint (checkpoint) tf.logging.info ('loading latest checkpoint # file: {}'.format (checkpoint) #model_saver.restore (SES) S, checkpoint, #newstyle) = np.zeros ([num_styles], dtype=np.float32) #newstyle[18] = 0.5 #newstyle[17] = 0.5 (#newImage = np.zeros (1, imgWidth, imgHeight, channel) #style_image) = transform.eval (feed_dict={inputImage:newImage, styles:newstyle}) #style_image = style_image[0] #imshow (style_image) (#pyplot.show)

Here the input node is input and style, the output node is model in transformer/expand/conv3/conv/Sigmoid.

To save the model calculation in the local folder. Next is the
parameter map and CKPT in the merger, and generates a mobile terminal can use the Pb file, this step can refer to my previous article “iOS+Tensorflow image recognition”, is very easy to achieve.

IOS project


In the above preparation work, if you have already set up the iOS+TF project according to the procedure, you need to import the final Pb file. Engineering structure as shown in figure:

IOS implementation of class Prisma software
XCode project

Then use the Pb file in iOS, I am here directly into the Google provided by the tensorflow_utils, the use of this class inside the LoadModel method can quickly generate session.

- (void) viewDidLoad [super tensorflow:: Status {viewDidLoad]; load_status; load_status = LoadModel (@ @ "rounded_graph", "Pb", & tf_session); if (load_status.ok) (!) {LOG (FATAL) < < "Couldn't load model:" < < load_status; currentStyle = 0} isDone; [UIColor = true; _styleImageView.layer.borderColor = grayColor].CGColor; _styleImageView.layer.borderWidth = 0.5; _ogImageView.layer.borderColor = [UIColor grayColor].CGColor; _ogImageView.layer.borderWidth = 0.5;}

The last is to get the picture, the implementation of the operation, the creation of art image display. Here the picture needs to be converted into bitmap and then get data value, showing the picture is also the process of understanding. Specific code is as follows:

- (void) runCnn: (UIImage * compressedImg) {unsigned char *pixels = [self getImagePixel:compressedImg]; int image_channels = 4; tensorflow:: Tensor image_tensor (tensorflow:: DT_FLOAT, tensorflow:: TensorShape ({1, wanted_input_height, wanted_input_width, wanted_input_channels})); auto image_tensor_mapped = image_tensor.tensor< float, 4> (tensorflow:); uint8: *in = pixels float; *out = image_tensor_mapped.data; for (int) (y = 0; y < wanted_input_height; ++y) {float = out + *out_row (y * wanted_input_width * wanted_input_channels); for (int x = 0; X < wanted_inp Ut_width; ++x) {tensorflow:: uint8 = in + *in_pixel (x * wanted_input_width * image_channels) + (y * image_channels); float *out_pixel = out_row + (x * wanted_input_channels); for (int c = 0; C < wanted_input_channels; ++c) {out_pixel[c] = in_pixel[c];}}} (tensorflow:: Tensor style tensorflow:: DT_FLOAT, tensorflow:: TensorShape ({32})); float *style_data = style.tensor< float, 1> (.Data) (memset); (style_data, 0, sizeof (style_data) * 32); style_data[currentStyle] = 1; if ((tf_session.get)) {std:: vector< tensorflow:: Tensor> outputs; tensorflow: Status run_status (Run = tf_session-> {{contentNode, image_tensor}, {styleNode, style}}, {outputNode}, & outputs) {}; if (run_status.ok) (!) {LOG (ERROR) < < "Running model failed:" < < run_status; isDone = true; free (pixels);} else {float *styledData = outputs[0].tensor< float, 4> (());.Data; UIImage *styledImg = [self createImage:styledData]; dispatch_async (dispatch_get_main_queue), _styleImag (^{ EView.image = styledImg; dispatch_after (dispatch_time (DISPATCH_TIME_NOW (int64_t) (0.3 * NSEC_PER_SEC)), dispatch_get_main_queue (isDone), ^{= true; free (pixels);};};}}))} - (unsigned * char) getImagePixel: (UIImage * image) {int width = image.size.width; int = height image.size.height; CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB (unsigned); char = *rawData (unsigned char*) calloc (height * width * 4, sizeof (unsigned char)); NSUInteger bytesPerPixel = 4; NSUInteger = bytesPerRow bytesPerPixel * width NSUInteger CGContextRef; bitsPerComponent = 8; context = CGBitmapContextCreate (rawData, width, Hei Ght, bitsPerComponent, bytesPerRow, colorSpace, kCGImageAlphaPremultipliedLast |); kCGBitmapByteOrder32Big CGColorSpaceRelease (colorSpace); CGContextDrawImage (context, CGRectMake (0, 0, width, height), image.CGImage); UIImage *ogImg = [UIImage imageWithCGImage:CGBitmapContextCreateImage (context)]; dispatch_async (dispatch_get_main_queue), ^{(_ogImageView.image = ogImg;}); CGContextRelease (context) return; rawData;} - (UIImage *) createImage: (float * pixels) {unsigned char = *rawData (unsigned char*) calloc (wanted_input_height * wanted_input_width * 4, sizeof (unsigned char) for (int); y = 0; y < Wanted_input_height; unsigned; ++y) {char = rawData + *out_row (y * wanted_input_width * 4); for (int x = 0; X < wanted_input_width; ++x) {float = pixels + *in_pixel (x * wanted_input_width * 3) + (y * 3); unsigned char (*out_pixel = out_row + X * 4 for (int); C = 0; C < wanted_input_channels; ++c) {out_pixel[c]} * 255 = in_pixel[c]; out_pixel[3] = UINT8_MAX;}} CGColorSpaceRef (NSUInteger) colorSpace = CGColorSpaceCreateDeviceRGB; bytesPerPixel = 4; NSUInteger bytesPerRow = bytesPerPixel * wanted_input_width; NSUInteger bitsPerComponent = 8; CGContextRef = CGBitmapContextCreate (rawData context Wanted_input_width, wanted_input_height, bitsPerComponent, bytesPerRow, colorSpace, kCGImageAlphaPremultipliedLast, kCGBitmapByteOrder32Big |); CGColorSpaceRelease (colorSpace); UIImage *retImg = [UIImage imageWithCGImage: CGBitmapContextCreateImage (context)]; CGContextRelease (context); free (rawData); return retImg;}

Note that here in front of the python project has been defined, 512 512 is my input and output the size of the image.

Connect the iPhone, run the project ^_^


Finally connected to the phone to run, you can create their own art class pictures. ?

Put a few running effect diagram:

IOS implementation of class Prisma software
screenshot 1
IOS implementation of class Prisma software
screenshot 2
IOS implementation of class Prisma software
screenshot 3