Output Stylized Image looks different on torch and coreml

[engahmed1190]

I have used same image on both models , Torch model present a good output while the Coreml model changes the main picture color .

i used this image for testing

the output from torch model is

Coreml model provides this

the model trained options

percep_loss_weight: 1	
padding_type: reflect-start	
batch_size: 4	
arch: c9s1-32,d64,d128,R128,R128,R128,R128,R128,u64,u32,c9s1-3	
resume_from_checkpoint: 	
style_image: images/styles/home.jpeg	
max_train: -1	
style_image_size: 700	
tv_strength: 0.001	
lr_decay_factor: 0.5	
checkpoint_name: checkpoint	
loss_network: models/vgg16.t7	
gpu: 0	
content_layers: 16	
task: style	
use_instance_norm: 1	
tanh_constant: 150	
preprocessing: vgg	
style_weights: 5,5,5,5	
checkpoint_every: 1000	
num_val_batches: 10	
num_iterations: 10000	
use_cudnn: 1	
pixel_loss_weight: 0	
content_weights: 1	
style_target_type: gram	
weight_decay: 0	
pixel_loss_type: L2	
lr_decay_every: -1	
learning_rate: 0.001	
style_layers: 4,9,16,23	
backend: cuda	
upsample_factor: 4	
h5_file: /home/Desktop/coreml/models/file.h5	

[engahmed1190]

my first thoughts that it's related to this part of the code

if is_grayscale:
        gray_bias = deprocessing_args.get('gray_bias', 0.0)
        W = np.array([image_scale])
        b = np.array([gray_bias])
    else:
        W = np.array([image_scale, image_scale, image_scale])

        red_bias = deprocessing_args.get('red_bias', 0.0)
        green_bias = deprocessing_args.get('green_bias', 0.0)
blue_bias = deprocessing_args.get('blue_bias', 0.0) 

[opedge]

@engahmed1190 I can try to investigate and debug this problem if you could attach your torch7 model.

[engahmed1190]

here is the torch model
Model Link

[opedge]

@engahmed1190 I tried to run the model which you provided with latest torch7 installation (without CUDA on macOS) using this command: th fast_neural_style.lua -model home.jpeg_700.t7 -input_image test.jpg -output_image test_result.jpg

I got this output:

So it looks like your output from CoreML. May be something inconsistent with torch7 installation or with fast-neural-style repo revision that you use.

[engahmed1190]

i have tried the same model but different result
can you please try the same model but with batch normalization , it works with coreml in batch normalization
Batch Model

[engahmed1190]

can you suggest a solution for this case @opedge
seems to be working with batch norm and instance norm getting reddish

[gbildson]

I've noticed that all new instance norm models don't work when converted to coreml. Also, all historical models in fast-neural-style don't work in torch. Have yet to figure out the connection.

[gbildson]

I found an older machine with an older torch setup (~ 6 months) and the server images for old models generate fine. I then stepped through the updates that I had applied to the newer machine.

luarocks install torch
luarocks install nn

The server started screwing up image generation for old models after applying the "luarocks install nn" step.

This makes sense I believe because the pytorch convert step always screws up on new models and I suspect it has something to do with the use of "from torch.nn import InstanceNorm3d".

[AndreJFBico]

Ive also been having issues with instance normalization models that i have trained in torch and converted to coreml.

Even training with the exact same parameters as the johnson pretrained models it causes issues.

Im trying now with batch normalization.

[gbildson]

Yeah, batch works but lower quality and/or bigger model.

[opedge]

For appropriate conversion may be you need to implement InstanceNorm layer by yourself like I did in sample converter but using another algorithms than InstanceNorm3d layer from pytorch.

[AndreJFBico]

If your still interested about instance normalization i ended up using this pytorch https://github.com/abhiskk/fast-neural-style implementation, and its instance normalization works straight off the bat.

I adjusted the current torch2coreml implementation and adjusted it to convert the pretrained models present in the link above, directly from pytorch instead of legacy torch.

This is their instance normalization code.

class InstanceNormalization(torch.nn.Module):
    """InstanceNormalization
    Improves convergence of neural-style.
    ref: https://arxiv.org/pdf/1607.08022.pdf
    """

    def __init__(self, dim, eps=1e-9):
        super(InstanceNormalization, self).__init__()
        self.scale = nn.Parameter(torch.FloatTensor(dim))
        self.shift = nn.Parameter(torch.FloatTensor(dim))
        self.eps = eps
        self._reset_parameters()

    def _reset_parameters(self):
        self.scale.data.uniform_()
        self.shift.data.zero_()

    def forward(self, x):
        n = x.size(2) * x.size(3)
        t = x.view(x.size(0), x.size(1), n)
        mean = torch.mean(t, 2).unsqueeze(2).unsqueeze(3).expand_as(x)
        # Calculate the biased var. torch.var returns unbiased var
        var = torch.var(t, 2).unsqueeze(2).unsqueeze(3).expand_as(x) * ((n - 1) / float(n))
        scale_broadcast = self.scale.unsqueeze(1).unsqueeze(1).unsqueeze(0)
        scale_broadcast = scale_broadcast.expand_as(x)
        shift_broadcast = self.shift.unsqueeze(1).unsqueeze(1).unsqueeze(0)
        shift_broadcast = shift_broadcast.expand_as(x)
        out = (x - mean) / torch.sqrt(var + self.eps)
        out = out * scale_broadcast + shift_broadcast
        return out

[opedge]

@AndreJFBico sounds great! May you'd like to make a PR for your pytorch models support and instance_norm layer?

[RahulBhalley]

@AndreJFBico can you please provide the modified torch2coreml code for direct conversion from PyTorch pre-trained model to CoreML?