Skip to main content
Springer Nature Link
Log in

Multimode-fused reservoir computing based on CH3NH3PbI3 nanowire optoelectronic memristor for spatiotemporal information processing

  • Research Paper
  • Published:
Save article
View saved research
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Multiple sensing and processing capabilities enable an organism to obtain comprehensive information and make real-time decisions in unstructured environments. Inspired by the biological counterpart, developing neuromorphic hardware with multimodal fusion is supposed to realize high-precision processing for complex information. In this work, we developed a visual-auditory-fused reservoir computing system by combining a CH3NH3PbI3 nanowire/pentacene optoelectronic memristor and a sound sensor. Under the stimulation of optical/electrical signals, the CH3NH3PbI3 nanowire-based device exhibited volatility under optical or electrical stimulation, providing an essential foundation for visual-auditory-fused reservoir computing. The motion detection function was demonstrated using synchronous visual and auditory sensory information. Compared with single sensory input, the detection accuracy increased from 51.2% to 95.5% by combining the visual and auditory modalities. The results presented in this work provide a promising strategy to develop efficient multimodal neuromorphic hardware for human-robot interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from €37.37 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price includes VAT (Netherlands)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

References

  1. Duan S, Shi Q, Wu J. Multimodal sensors and ML-based data fusion for advanced robots. Adv Intell Syst, 2022, 4: 2200213

    Article  Google Scholar 

  2. Tan H, Zhou Y, Tao Q, et al. Bioinspired multisensory neural network with crossmodal integration and recognition. Nat Commun, 2021, 12: 1120

    Article  Google Scholar 

  3. Jung Y H, Park B, Kim J U, et al. Bioinspired electronics for artificial sensory systems. Adv Mater, 2019, 31: 1803637

    Article  Google Scholar 

  4. Duan Q, Zhang T, Liu C, et al. Artificial multisensory neurons with fused haptic and temperature perception for multimodal in-sensor computing. Adv Intell Syst, 2022, 4: 2200039

    Article  Google Scholar 

  5. Zhu J, Zhang X, Wang R, et al. A heterogeneously integrated spiking neuron array for multimode-fused perception and object classification. Adv Mater, 2022, 34: 2200481

    Article  Google Scholar 

  6. Franzen L, Delis I, de Sousa G, et al. Auditory information enhances post-sensory visual evidence during rapid multisensory decision-making. Nat Commun, 2020, 11: 5440

    Article  Google Scholar 

  7. Wang Y, Cai Y, Wang F, et al. A three-dimensional neuromorphic photosensor array for nonvolatile in-sensor computing. Nano Lett, 2023, 23: 4524–4532

    Article  Google Scholar 

  8. Wang T Y, Meng J L, Li Q X, et al. Reconfigurable optoelectronic memristor for in-sensor computing applications. Nano Energy, 2021, 89: 106291

    Article  Google Scholar 

  9. Yang C, Wang H, Zhou G, et al. A multimodal perception-enabled flexible memristor with combined sensing-storage-memory functions for enhanced artificial injury recognition. Small, 2024, 20: 2402588

    Article  Google Scholar 

  10. Zhao L C, Zou H X, Wei K X, et al. Mechanical intelligent energy harvesting: from methodology to applications. Adv Energy Mater, 2023, 13: 2300557

    Article  Google Scholar 

  11. Yu J, Wang Y, Qin S, et al. Bioinspired interactive neuromorphic devices. Mater Today, 2022, 60: 158–182

    Article  Google Scholar 

  12. Midya R, Wang Z, Asapu S, et al. Reservoir computing using diffusive memristors. Adv Intell Syst, 2019, 1: 1900084

    Article  Google Scholar 

  13. Fang H, Ma S, Wang J, et al. Multimodal in-sensor computing implemented by easily-fabricated oxide-heterojunction optoelectronic synapses. Adv Funct Mater, 2024, 34: 2409045

    Article  Google Scholar 

  14. Ma S, Zhou Y, Wan T, et al. Bioinspired in-sensor multimodal fusion for enhanced spatial and spatiotemporal association. Nano Lett, 2024, 24: 7091–7099

    Article  Google Scholar 

  15. Sun L, Wang Z, Jiang J, et al. In-sensor reservoir computing for language learning via two-dimensional memristors. Sci Adv, 2021, 7: eabg1455

    Article  Google Scholar 

  16. Ji X, Dong Z, Han Y, et al. A brain-inspired hierarchical interactive in-memory computing system and its application in video sentiment analysis. IEEE Trans Circ Syst Video Technol, 2023, 33: 7928–7942

    Article  Google Scholar 

  17. Ji X, Dong Z, Lai C S, et al. A brain-inspired in-memory computing system for neuronal communication via memristive circuits. IEEE Commun Mag, 2022, 60: 100–106

    Article  Google Scholar 

  18. Wang G, Wang R, Kong W, et al. Simulation of retinal ganglion cell response using fast independent component analysis. Cogn Neurodyn, 2018, 12: 615–624

    Article  Google Scholar 

  19. Yan X, He H, Liu G, et al. A robust memristor based on epitaxial vertically aligned nanostructured BaTiO3-CeO2 films on silicon. Adv Mater, 2022, 34: 2110343

    Article  Google Scholar 

  20. Gao S, Liu G, Yang H, et al. An oxide Schottky junction artificial optoelectronic synapse. ACS Nano, 2019, 13: 2634–2642

    Article  Google Scholar 

  21. Pan X, Wang J, Deng Z, et al. A memristor-based bioinspired multimodal sensory memory system for sensory adaptation of robots. Adv Intell Syst, 2022, 4: 2200031

    Article  Google Scholar 

  22. Yan X, Zhang Y, Fang Z, et al. A multimode-fused sensory memory system based on a robust self-assembly nanoscaffolded BaTiO3:Eu2O3 memristor. InfoMat, 2023, 5: e12429

    Article  Google Scholar 

  23. Guo J, Guo F, Zhao H, et al. In-sensor computing with visual-tactile perception enabled by mechano-optical artificial synapse. Adv Mater, 2025, 37: 2419405

    Article  MathSciNet  Google Scholar 

  24. Wang Y, Gong Y, Yang L, et al. MXene-ZnO memristor for multimodal in-sensor computing. Adv Funct Mater, 2021, 31: 2100144

    Article  Google Scholar 

  25. Kolarik A J, Moore B C J, Zahorik P, et al. Auditory distance perception in humans: a review of cues, development, neuronal bases, and effects of sensory loss. Atten Percept Psychophys, 2016, 78: 373–395

    Article  Google Scholar 

  26. Opoku-Baah C, Schoenhaut A M, Vassall S G, et al. Visual influences on auditory behavioral, neural, and perceptual processes: a review. J Assoc Res Oto, 2021, 22: 365–386

    Google Scholar 

  27. Ozker M, Schepers I M, Magnotti J F, et al. A double dissociation between anterior and posterior superior temporal gyrus for processing audiovisual speech demonstrated by electrocorticography. J Cogn Neurosci, 2017, 29: 1044–1060

    Article  Google Scholar 

  28. Bednarek M, Kicki P, Walas K. On robustness of multi-modal fusion-robotics perspective. Electronics, 2020, 9: 1152

    Article  Google Scholar 

  29. Song Y, Li Z, Li G, et al. Multi-sensory visual-auditory fusion of wearable navigation assistance for people with impaired vision. IEEE Trans Automat Sci Eng, 2025, 22: 4724–4736

    Article  Google Scholar 

  30. Chen Q, De Marco N, Yang Y M, et al. Under the spotlight: the organic-inorganic hybrid halide perovskite for optoelectronic applications. Nano Today, 2015, 10: 355–396

    Article  Google Scholar 

  31. Liu L, Ma Y, Wang Y, et al. Hole-transport management enables 23%-efficient and stable inverted perovskite solar cells with 84% fill factor. Nano-Micro Lett, 2023, 15: 117

    Article  Google Scholar 

  32. Wu D, Xu Y, Zhou H, et al. Ultrasensitive, flexible perovskite nanowire photodetectors with long-term stability exceeding 5000 h. InfoMat, 2022, 4: e12320

    Article  Google Scholar 

  33. Zhang Y, Ma Z, Chen Z, et al. Perovskite-nanowire-array-based continuous-state programmable artificial synapse for neuromorphic computing. Adv Intell Syst, 2024, 6: 2300586

    Article  Google Scholar 

  34. Feng J, Gong C, Gao H, et al. Single-crystalline layered metal-halide perovskite nanowires for ultrasensitive photodetectors. Nat Electron, 2018, 1: 404–410

    Article  Google Scholar 

  35. Im J H, Luo J, Franckevičius M, et al. Nanowire perovskite solar cell. Nano Lett, 2015, 15: 2120–2126

    Article  Google Scholar 

  36. Long Z, Qiu X, Chan C L J, et al. A neuromorphic bionic eye with filter-free color vision using hemispherical perovskite nanowire array retina. Nat Commun, 2023, 14: 1972

    Article  Google Scholar 

  37. Wan C, Cai P, Wang M, et al. Artificial sensory memory. Adv Mater, 2020, 32: 1902434

    Article  Google Scholar 

  38. Choi C, Choi M K, Liu S, et al. Human eye-inspired soft optoelectronic device using high-density MoS2-graphene curved image sensor array. Nat Commun, 2017, 8: 1664

    Article  Google Scholar 

  39. Hudspeth A J. Integrating the active process of hair cells with cochlear function. Nat Rev Neurosci, 2014, 15: 600–614

    Article  Google Scholar 

  40. Dalhoff E, Turcanu D, Zenner H P, et al. Distortion product otoacoustic emissions measured as vibration on the eardrum of human subjects. Proc Natl Acad Sci USA, 2007, 104: 1546–1551

    Article  Google Scholar 

  41. Green A M, Angelaki D E. Multisensory integration: resolving sensory ambiguities to build novel representations. Curr Opin Neurobiol, 2010, 20: 353–360

    Article  Google Scholar 

  42. Zhu P, Gu S, Shen X, et al. Direct conversion of perovskite thin films into nanowires with kinetic control for flexible optoelectronic devices. Nano Lett, 2016, 16: 871–876

    Article  Google Scholar 

  43. Gao L, Zeng K, Guo J, et al. Passivated single-crystalline CH3NH3PbI3 nanowire photodetector with high detectivity and polarization sensitivity. Nano Lett, 2016, 16: 7446–7454

    Article  Google Scholar 

  44. Liu M, Johnston M B, Snaith H J. Efficient planar heterojunction perovskite solar cells by vapour deposition. Nature, 2013, 501: 395–398

    Article  Google Scholar 

  45. Frątczak E Z, Uznański P, Moneta M E. Characterization of molecular organization in pentacene thin films on SiO2 surface using infrared spectroscopy, spectroscopic ellipsometry, and atomic force microscopy. Chem Phys, 2015, 456: 49–56

    Article  Google Scholar 

  46. Hudgins D M, Allamandola L J. Infrared spectroscopy of matrix-isolated polycyclic aromatic hydrocarbon cations. 3. the polyacenes anthracene, tetracene, and pentacene. J Phys Chem, 1995, 99: 8978–8986

    Article  Google Scholar 

  47. Mori M, Abegg M H, Gähwiler B H, et al. A frequency-dependent switch from inhibition to excitation in a hippocampal unitary circuit. Nature, 2004, 431: 453–456

    Article  Google Scholar 

  48. Adermark L, Lovinger D M. Retrograde endocannabinoid signaling at striatal synapses requires a regulated postsynaptic release step. Proc Natl Acad Sci USA, 2007, 104: 20564–20569

    Article  Google Scholar 

  49. McDonald J J, Teder-Sälejärvi W A, Ward L M. Multisensory integration and crossmodal attention effects in the human brain. Science, 2001, 292: 1791

    Article  Google Scholar 

  50. Liu Q, Yin L, Zhao C, et al. Hybrid mixed-dimensional perovskite/metal-oxide heterojunction for all-in-one opto-electric artificial synapse and retinal-neuromorphic system. Nano Energy, 2022, 102: 107686

    Article  Google Scholar 

  51. Shan X, Zhao C, Lin Y, et al. Optoelectronic synaptic device based on ZnO/HfOx heterojunction for high-performance neuromorphic vision system. Appl Phys Lett, 2022, 121: 263501

    Article  Google Scholar 

  52. Lee M, Lee W, Choi S, et al. Brain-inspired photonic neuromorphic devices using photodynamic amorphous oxide semiconductors and their persistent photoconductivity. Adv Mater, 2017, 29: 1700951

    Article  Google Scholar 

  53. Lin Y, Zeng T, Xu H, et al. Transferable and flexible artificial memristive synapse based on WOx Schottky junction on arbitrary substrates. Adv Elect Mater, 2018, 4: 1800373

    Article  Google Scholar 

  54. Wang Z Q, Xu H Y, Li X H, et al. Synaptic learning and memory functions achieved using oxygen ion migration/diffusion in an amorphous InGaZnO memristor. Adv Funct Mater, 2012, 22: 2759–2765

    Article  Google Scholar 

  55. Lee Y, Oh J Y, Xu W, et al. Stretchable organic optoelectronic sensorimotor synapse. Sci Adv, 2018, 4: eaat7387

    Article  Google Scholar 

  56. Ji G, Zhao B, Song F, et al. The energy level alignment at the CH3NH3PbI3/pentacene interface. Appl Surf Sci, 2017, 393: 417–421

    Article  Google Scholar 

  57. Wang Y, Liu M, Fan F, et al. Enhanced full-spectrum photocatalytic activity of 3D carbon-coated C3N4 nanowires via giant interfacial electric field. Appl Catal B-Environ, 2022, 318: 121829

    Article  Google Scholar 

  58. Bizley J K, Jones G P, Town S M. Where are multisensory signals combined for perceptual decision-making? Curr Opin Neurobiol, 2016, 40: 31–37

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China for Distinguished Young Scholars (Grant No. 52025022), Funds from Jilin Province (Grant No. 20240210002GX), National Natural Science Foundation of China (Grant Nos. U23A20568, 52372137, 52272140, 52402175, 52472149), China Postdoctoral Science Foundation (Grant No. GZB20240135), “111” Project (Grant No. B25030), Fundamental Research Funds for the Central Universities (Grant No. 2412024Q17006), and Scientific Research Innovation Capability Support Project for Young Faculty (Grant No. ZYGXQNJSKYCXNLZCXM-I10).

Author information

Authors and Affiliations

  1. State Key Laboratory of Integrated Optoelectronics, Northeast Normal University, Changchun, 130024, China

    Yue Wang, Xuanyu Shan, Wei Liu, Jiahui Zheng, Xiaoning Zhao, Zhongqiang Wang, Ya Lin, Ye Tao, Haiyang Xu & Yichun Liu

  2. China FAW Group Co., Ltd., Changchun, 130011, China

    Qiang Wang, Zewei Wang & Siyu Liu

Authors
  1. Yue Wang
    View author publications

    Search author on:PubMed Google Scholar

  2. Xuanyu Shan
    View author publications

    Search author on:PubMed Google Scholar

  3. Qiang Wang
    View author publications

    Search author on:PubMed Google Scholar

  4. Zewei Wang
    View author publications

    Search author on:PubMed Google Scholar

  5. Siyu Liu
    View author publications

    Search author on:PubMed Google Scholar

  6. Wei Liu
    View author publications

    Search author on:PubMed Google Scholar

  7. Jiahui Zheng
    View author publications

    Search author on:PubMed Google Scholar

  8. Xiaoning Zhao
    View author publications

    Search author on:PubMed Google Scholar

  9. Zhongqiang Wang
    View author publications

    Search author on:PubMed Google Scholar

  10. Ya Lin
    View author publications

    Search author on:PubMed Google Scholar

  11. Ye Tao
    View author publications

    Search author on:PubMed Google Scholar

  12. Haiyang Xu
    View author publications

    Search author on:PubMed Google Scholar

  13. Yichun Liu
    View author publications

    Search author on:PubMed Google Scholar

Corresponding authors

Correspondence to Xuanyu Shan, Xiaoning Zhao or Zhongqiang Wang.

Additional information

Supporting information Figures S1–S7. The supporting information is available online at info.scichina.com and link.springer.com. The supporting materials are published as submitted, without typesetting or editing. The responsibility for scientific accuracy and content remains entirely with the authors.

Supplementary File

11432_2025_4643_MOESM1_ESM.pdf (download PDF )

Multimode-fused reservoir computing based on CH3NH3PbI3 nanowires optoelectronic memristor for spatiotemporal information processing

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Shan, X., Wang, Q. et al. Multimode-fused reservoir computing based on CH3NH3PbI3 nanowire optoelectronic memristor for spatiotemporal information processing. Sci. China Inf. Sci. 69, 142403 (2026). https://doi.org/10.1007/s11432-025-4643-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Version of record:

  • DOI: https://doi.org/10.1007/s11432-025-4643-6

Keywords