INT TFLITE very much slower than FLOAT TFLITE #21698
Description
Hi Guys. I am using Mobilenet 0.25,128. I used the pretrained models provided in the repo for obtaining the int tflite and float tflite models for the same found here. I am trying to infer some images. Using the imagenet images and some other test images as well, the FLOAT TFLITE is faster than the INT TFLITE (FLOAT TFLITE takes roughly 3-4 milliseconds while INT one takes 8-9 ms). Any suggestions as to why this is happening ?I am running the inferences using the tflite interpreter following the documentation here. This issue happens with the tf-nightly builds as well as the normal tensorflow. Tried on both, and also on both python 2 as well as 3.
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow):Yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Ubuntu
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
- TensorFlow installed from (source or binary):Binary
- TensorFlow version (use command below):1.10 (tf-nightly)
- Python version:2
- Bazel version (if compiling from source)
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version: No
- GPU model and memory: No
Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Source code / logs
####INT TFLITE CODE####
#Load TFLite model and allocate tensors. Select the appropriate model and give its path
interpreter = tf.contrib.lite.Interpreter(model_path='mobilenet_v1_0.25_128_quant.tflite')
#For allocating the model tensors
interpreter.allocate_tensors()
#Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
#print(input_details)
#print(output_details)
k=1
for image_path in TEST_IMAGE_PATHS:
#Loading the image and resizing into the correct shape
#Change the .astype(np.uint8) (used when using int tflite) to .astype(np.float32) if using float tflite
img = np.array(PIL.Image.open(image_path).resize((128, 128))).astype(np.uint8)
img = img.reshape(1,128,128,3)
#print(input_details)
#print (img.shape)
#Setting the input image to the input tensor
interpreter.set_tensor(input_details[0]['index'], img)
#Running inference and timing it
start = time.time()
interpreter.invoke()
end = time.time()
#Getting the output information
output_data = interpreter.get_tensor(output_details[0]['index'])
#Converting the prediction into human readable form
#label_map = imagenet.create_readable_names_for_imagenet_labels()
print("Top 1 Prediction: ", output_data.argmax()-1, output_data.max(), k+1,labels[output_data.argmax()-1])
total_infer_time+=(end-start)
k+=1
#printing average inference time along with the accuracy
print("Total infer time avg (in seconds) == ",total_infer_time/no_images)
####FLOAT TFLITE CODE####
#Load TFLite model and allocate tensors. Select the appropriate model and give its path
interpreter = tf.contrib.lite.Interpreter(model_path='mobilenet_v1_0.25_128.tflite')
#For allocating the model tensors
interpreter.allocate_tensors()
#Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
#print(input_details)
#print(output_details)
k = 1
for image_path in TEST_IMAGE_PATHS:
print image_path
#Loading the image and resizing into the correct shape
#Change the .astype(np.uint8) (used when using int tflite) to .astype(np.float32) if using float tflite
img = np.array(PIL.Image.open(image_path).resize((128, 128))).astype(np.float32) / 128 - 1
img = img.reshape(1,128,128,3)
#print (img.shape)
#Setting the input image to the input tensor
interpreter.set_tensor(input_details[0]['index'], img)
#Running inference and timing it
start = time.time()
interpreter.invoke()
end = time.time()
#Getting the output information
output_data = interpreter.get_tensor(output_details[0]['index'])
#Converting the prediction into human readable form
#label_map = imagenet.create_readable_names_for_imagenet_labels()
#print("Top 1 Prediction: ", output_data.argmax()-1, output_data.max(), k+1,labels[output_data.argmax()-1])
#detect_dog(output_data)
'''
#For accuracy
number = []
number = int(mobilenet_labels[k])
if(number==output_data.argmax()):
accuracy+=1
'''
total_infer_time+=(end-start)
print total_infer_time/(k+1)
k+=1
#printing average inference time along with the accuracy
print("Total infer time avg (in seconds) == ",total_infer_time/no_images)