Question Data Processing

[mdaehl]

Hello, in your released preprocessed data there are already gt semantic maps available for the different scenes. I cannot find these in the original datasets.
How did you get these? Were they labeled by you afterwards?

[InhwanBae]

Hi @mdaehl,

Thanks for your interest in our work! The semantic maps we use are not ground-truth maps and therefore are not included in the original datasets.

To generate semantic maps for each dataset scene, we used the pretrained Grounded SAM 2 (grounded_sam2_local_demo.py). I experimented with several alternatives, including satellite segmentation models, pretrained SegFormer, pretrained segmentation models in the Y-Net trajectory predictor, and our trial models trained from ground-truth segmentation labels. But we found that Grounded SAM 2 produced the most reliable results.

Y-Net pretrained	Grounded SAM 2	Grounded SAM 2 + Postprocess

For Grounded SAM 2, we provide text labels, for example,
text_prompt = "sidewalk. pavement. road. driveway. grass. lawn. terrain. bush. shrub. tree. structure. building. house. wall. car. vehicle. background." We tested various text prompts and selected the most effective one for each dataset. Afterward, we removed dynamic objects like pedestrians, bicycles and vehicles, and then computed a weighted sum of all masks across 7 classes. Lastly, we assigned each pixel to the class with the maximum response to get the final semantic segmentation map.

If you have any further questions, feel free to reach out!

[mdaehl]

Thank you so much for the quick and detailed answer @InhwanBae ! In your paper and also in the README you present a cool example of the evaluation on New York City. Could you please explain how you were able to create these? It would be interesting to know how you both created the input for the model (BEV image) and how you created the respective 3D model for visualization.

[InhwanBae]

To apply our CrowdES framework to additional scenes such as New York City, you can use the Custom Input option. Since the README provides only limited details, I explain the process here more thoroughly.

1. Crowd Behavior Generation

To test CrowdES on scenes beyond the provided evaluation datasets, you need to add custom data under the datasets/Synthetic/ directory. Specifically, populate the following subfolders:

• image_terrain/: contains the BEV background image. In my experiments, I used satellite or map images from Google Maps.
• homography/: contains a text file specifying the scale, i.e., how many meters correspond to one pixel in the image.
• gt/: contains maps of walkable regions, which Google Maps provides as layers. These maps include appearance density and population density maps.

The folder structure should look like this:

datasets/Synthetic
├─ image_terrain
│  └─ manhattan_bg.png
├─ homography
│  └─ manhattan_scale.txt
└─ gt
   ├─ manhattan_appearance_density.png
   └─ manhattan_population_density.png

Note

For convenience, I share the real-world datasets.zip file that I used in writing the paper. Since the task involves large-scale outdoor environments, I used the SDD pretrained weights in the paper.

2. 3D Visualization

Setting up 3D visualization for custom real-world scenes is somewhat tricky and may take about a day initially (after that, visualizing 3D maps becomes surprisingly easier).

Note

Due to potential license issues with UE4, I cannot publicly upload the files. If you encounter difficulties following this process, feel free to reach out to me via the personal email address listed in the paper, and I will share the full UnrealEngine4 project files.

2.1. Extracting Maps from Google Earth

Option 1: Using Microsoft Flight Simulator SDK (new)
First, download the Google Earth Decoder and patch files for MSFS. Next, select the area to capture the 3D maps for download. Then, import the captured .gltf data into Blender, select all meshes, and click File → External Data → Find Missing Files to add textures. Finally, export the processed 3D map as an FBX file from Blender. You should embed all textures into a single FBX file by changing the Path Mode to Copy and clicking the Embed Textures icon. A useful tutorial can be found here: (YouTube) Import Google map 3d models into your 3d software

Option 2: Using RenderDoc
First, use RenderDoc to capture 3D maps from Google Earth. Then, import the captured .rdc data into Blender 4.1 using MapsModelsImporter. Finally, export the processed 3D map as an FBX file from Blender. You should embed all textures into a single FBX file by changing the Path Mode to Copy and clicking the Embed Textures icon. A useful tutorial can be found here: (YouTube) Google Maps to Blender.

Tip: It would be better to merge all fragmented meshes by pressing Ctrl + J in Blender to make it easier to apply textures in Unreal Engine.

2.2. Building UnrealEngine-CARLA

Next, launch the CARLA map editor. Follow the instructions to compile UnrealEngine-CARLA. (I used UE4 version, but UE5 will also fine) After compilation, you will be able to access the Unreal Engine map development interface, as shown below:

git clone -b 0.9.15.2 https://github.com/carla-simulator/carla.git carla_0.9.15.2
git clone --depth 1 -b carla https://github.com/CarlaUnreal/UnrealEngine.git carla_ue4

2.3. Loading the 3D Map and Launching Visualization

Start by duplicating an existing Town (e.g., Town 1). Create a plane high enough so that it is not interfered with by buildings, set Collide = True, and apply the BEV image as a texture. Adjust the scale so that the map dimensions match real-world measurements. Then, import the FBX model exported from Blender, and align its size and position with the plane. (Optional) Set the FBX model to Collide = False and make the plane transparent for better visualization. Make sure to record the origin x, y coordinates of the plane.

Finally, for visualizing crowd trajectories in 3D, use the scripts provided in the 3D Visualization Toolkit. Update the parameters origin(top-left meter coordinates in Unreal Engine) and multi(pixel-to-meter ratio) in the script according to your new scene. See the relevant code snippet here:

Crowd-Behavior-Generation/3D Visualization Toolkit/run_trajsim.py

Lines 637 to 657 in 299ea7b

	if args.scene == 'hyang':
	# hyang
	# origin, multi, swap_xy, flip_x, flip_y = np.array([[119.2, 224.7]]), 1 / 29.17, False, False, False
	origin, multi, swap_xy, flip_x, flip_y = np.array([[120.2, 223.3]]), 1 / 29.17, False, False, False #shifted
	client.load_world('CrowdES_hyang')
	z_default=1.5
	elif args.scene == 'zara':
	origin, multi, swap_xy, flip_x, flip_y = np.array([[315 - 15.6535488*0.5*1.3, 215 - 12.52283904*0.5*1.3 + 0.5]]), 0.02174104*1.3, False, False, False # zara 15.6535488 12.52283904
	client.load_world('CrowdES_zara')
	z_default=2.5
	# set camera
	world = client.get_world()
	transform = world.get_spectator().get_transform()
	transform.location = carla.Location(x=315.328552, y=220.958313, z=7.827725)
	transform.rotation = carla.Rotation(pitch=-51.071339, yaw=-90.952713, roll=0.000014)
	world.get_spectator().set_transform(transform)
	else:
	raise NotImplementedError