InDesign only produces a lo-res proxy for the image.
Try setting your View>Display Perfromance to High Quality.
It's hard to see what's going wrong with the image as you've used a JPEG which added compression - so both images look the same.
PNG is better for the screen than JPEG - can you upload PNG screen shots of your artwork?
Also - please note that Smart Vector objects are only vector inside Photoshop and outputting (printing) from PDF.
All smart objects, no matter what file format are always rasterised when insert to another document.
The only objects that can retain vector data is Vector Masks, Vector Shapes, and Text Layers (effects will always be rastered).
And the only editable supported file for files with these elements is an Photoshop PDF.
Note - all smart objects will always be rasterised when inserted to any program to output, smart vector objects only retain their vectors for scaling in photoshop - they are rasterised on output.
For this reason you should use Adobe Illustrator.