It probably has to do with the for loops. I'm not 100% sure why the gen function is this slow. I took only 10 images to send through the gen function you built, and it took about 8 minutes to run. I tried creating some dummy images and a much smaller model (shown below) to try and compare performances. When I was building up my data pipeline, the Tensorflow docs were very insistent that generators are unsafe for multiprocessing, and that the best way to build up a multiprocessing streaming pipeline is to extend. #validation_data=(, val_y)īut it is to slow, to much i think, so i would like to know if you know how to make it perform faster, my dataset is about 44k pictures and a pandas whit 44k rows whit 2 features, i hope you can helpme Output_liq =Dense(1,name="output", activation='sigmoid')(x3) This is the generator def batch_generator(df, path, position, batch_size):Īnd my model is this one input_data = Input(name="input_numerical",shape=(2,1,)) So what are Data Generators or Image Data Generators Essentially, it is a class under Keras which is very useful in the field of image processing. LiqPaths = sorted(list(glob.glob(basePath))) # read one or more samples from your storage, do pre-processing, etc.īasePath = os.() So my generator looks like this def load_data(df,path,position): Keras open images and create a batch with them I made a datagenerator for a multi input model using this two references Setup In memory data Basic preprocessing Mixed data types Using tf.
0 Comments
Leave a Reply. |