plot gradient with three dimensions (coded as red, green & blue) in a ternary plot (ggtern)

Question

I have the following data frame;

df <- data.frame(expand.grid(A = seq(0,1,0.1), B = seq(0,1,0.1), C = seq(0,1,0.1)))
df <- df[rowSums(df[,1:3]) == 1,]
df <- data.frame(df, value_A = 0.2 * df$A + -0.4 * df$B + 0 * df$C, value_B = 0.8 * df$A + 0.1 * df$B + 0.5 * df$C, value_C = 0 * df$A + 0.8 * df$B + 0 * df$C)
df[,c(4:6)] <- (df[,c(4:6)] + abs(apply(df[,c(4:6)],1,min))) / (apply(df[,c(4:6)],1,max) + abs(apply(df[,c(4:6)],1,min)))

df <- data.frame(df, color = rgb(red=df$value_A, green=df$value_B, blue=df$value_C))

In the data-frame, each row gives me a proportion of A's, B's and C's that sum up to one. Further, for each A, B and C, I have a value.

From these three values I generate one RGB value that gives me an indication of the relative importance of A,B & C under different proportions of A, B and C's.

Now I would like to plot these RGB values as a surface in a ternary plot.

I can plot one "dimension" with the following code using the ggtern package in R;

library(ggtern)

ggtern(df, aes(A,B,C, value=value_A)) + 
theme_showarrows() + 
stat_interpolate_tern(geom="polygon",
                     formula=value~x+y,
                     n=20, method='lm',
                     breaks=seq(0,1, by=0.001),
                     aes(fill=..level..), expand=F
) +
scale_fill_gradient(low="red", high="green") 

But instead of value_A, I actually want to use the RGB values directly.

However, I can not figure out, how, instead of value=value_A I can specify the color value for each point from which the surface is calculated directly.

Is this possible with ggplot2 / ggtern?