Summary

• Authors
• Abstract

Summary of Intro (make short and highlight the most important messages)

• What’s the problem?
• What do we introduce?
• How does it roughly work? (main differentiator to existing methods? What’s unique?)
• What can we do now?
• What do we show?
• What’s the impact?

Main Content

1. Introduction
   1. Motivation (why does this work matter?)
   2. Limitations of Existing Work (what can’t we achieve yet)
   3. Our Objective
   4. Technical Challenge
   5. Impact/Advantages/Contributions
2. Related Work
   • 3D Scene Representation
   • Text-guided 3D Object Generation
   • Text-guided 3D Scene Editing
3. Preliminary
   • 3DGS
   • LERF in segmenting 3D scenes
   • Zero123 in handling view consistency and object rendering
4. Methods

   • Overview

     • (Add a figure of the overall pipeline, and in-between results. Also show a figure of the capture system.)
     • (Describe the overall system and how they relate to each other.)
     • Global problem definition. Input, output, intermediate output.
     • Note: to do insertion, we always do removal first and fix artifacts and then generate and insert.

   • Detection and Segmentation of Removing Object.

     • We write as: during removal, we just delete all points of interest. We don’t compute a bounding box.

     • We train LERF on the pre-trained scene. Then we extract confidence score over 3D samples along rays.

     • Issues are

       • 3D segmentation is view-dependent
       • segmentation is not clean — boundary of object has similar confidence score to background (re-phrase, see image below)
       • multiple instances may exist (see image below).

       

       

     • Analysis

       • Default 3D bounding box segmentation in LERF is not robust enough. The original way is to pick raw_relevancy>0.5 for any positive query.
       • This results in overlarge bounding box and inclusion of background pixels (whose language features are not trained thoroughly).

     • Our solutions are

       • First filter out 10th percentile LERF sampling.
       • Then, set eps=0.001 in DBSCAN and get the largest cluster.
       • The two hyper parameters above are the most stable after tuning. For example, positive phrase is “pumpkin”. However, it is still not stable enough — stability depends on 1) view direction (in general, bird view with little background is more stable), 2) coarse and fine rendering (coarse gives better bbox mean and find gives better bbox scale).
       • Therefore, we pick coarse (low-resolution) sampling to estimate bbox mean an scale in order to determine oriented bbox to be removed.

   • Occlusion-aware Optimization for Removal

     • For removal step only, to resolve artifact remained from removal due to occlusion (touching surfaces), we adopted an off-the-shelf inpainting module (Haiwei’s GAN). We iteratively pick a random camera view around bbox to render. Then, we project bbox into the view and call the inpainting module. During optimization of 3DGS, we freeze points locations (set xyz, rotation and scale lr to 0), and only allow optimization of color features.

   • Illumination-aware Inpainting

     • We pick a camera view around bbox to render, where the camera view has the least occlusion. We detect by shooting rays and check if any non-zero opacity samples is prior to hitting bbox.

     • Then, we leverage the insight where 2D diffusion model inpaints image with awareness of scene — global illumination. Note that previous bbox is dilated and given to stable diffusion as guidance. Now, we call SAM to segment the object at the center. Inpainted result with background examples are as following.

       

       

   • Multi-view Consistent Generation in 3D Gaussian Splatting Representation

     • We input segmented and centered single-view image into Zero123 to optimize a multi-view consistent object with continuous illumination. The optimized object is also in 3DGS representation.
     • We further add VSD for supervision on texture only.
5. Results
   • (Illustrate effectiveness of the method and superiority over existing methods)
   • (evaluation of proposed system/approach)
   • Qualitative results - show the quality/ rendering results
     • [optional] Robustness of segmentation and removing bbox according to $t_{remove}$. But this is mainly the same as LERF.
     • Removal result with 2D inpainting
       • Original scene x 1
       • Random testing view of whole scene with removal x 3
     • Insertion result
       • Original scene x 1
       • Conditioned 2D inpainted single-view image x 1
       • Random 360deg view of generated 3D content x 2
       • Random testing view of whole scene with insertion x 3
   • Evaluation
     • Comparison with DreamEditor and Instruct-NeRF2NeRF. Other works are either arxiv or do not have released code.
       • DreamEditor
       • Instruct-NeRF2NeRF
   • Ablation
     • NeRFacto (best NeRF) vs. 3DGS
     • NeRF-based AIGC (speed and quality) vs. 3DGS-based AIGC
   • Application
6. Conclusion/Discussion
   • Summary of Achievements, and what is possible now that isn’t possible before (What are the findings? What is surprising? What have we learned?)
   • What are the advantages over existing methods (What are big questions and how do we get closer to it?)
7. Future Work
   • Failure cases
     • Due to non-robust Zero123.
     • [optional] Due to non-robust 3DGS.
     • Inconsistent/wrong illumination.
     • Lack of sharpness in texture.